Weekly Findings In Data Science and AI

Extracting Training Data From Neural Networks — Weekly Findings

Hidden gorillas, extracting training data from neural networks and a search engine for your life. These are topics included in this week’s findings.

The Gorilla In The Data

I found this amusing scientific experiment, where researchers asked students to investigate a dataset with information about BMI, gender and the number of steps taken.

The students were divided into two groups. The first group was tasked with analyzing three hypotheses and also asked if they could conclude anything else from the data. The second group was only asked what they could conclude from the dataset.

The catch? The data was completely artificial and there was a hidden waving gorilla in the dataset that you could find by plotting the number of steps against the BMI, see here for the plot of the gorilla. Thus, the real experiment was to see how many students would find the primate in each group.

The results showed that 5 out of 19 of the first group (analyzing hypothesis) found the gorilla but those with no hypothesis found it in 9 out of 14 cases. Thus, in this case, focusing on hypotheses clearly became a distraction in finding the real underlying pattern.

While the focus of the paper is science and hypothesis testing, I think this phenomenon is relevant to data science in general. It is easy to immediately turn to modeling or the task at hand before doing sufficient exploratory data analysis (EDA). While it may work perfectly fine to do so for “nice” datasets, with proper EDA you are more likely to spot problems, helpful patterns and in general gain a better understanding of the data.

In this instance, visualization in particular was the key technique for finding the gorilla. Many relationships are difficult to explain adequately with only numbers.

Check out the original paper:

Training Data Reconstruction From Neural Networks

This is a recent fascinating paper proposing a novel method for reconstructing training data from neural networks. That is, instead of going from data to model, we go from model to data. The results look promising:

The method is currently only for binary classifiers and the models are at a small scale. It is also not clear what samples can be recontructed, e.g. if it’s only the most important samples for the model. It will be interesting to see future developments in the area.

The paper inspires many new questions about how neural networks work but also vulnerabilities of models today. The authors mention the consequence for privacy and the potential of a so-called training-data reconstruction attack to extract private sensitive data from models, e.g. medical records.

The solution I could think of for preventing these types of attacks would be to augment or hash the training data, remove the original data and then only use the augmented data for fitting the model. This would mean that only the augmented data would be reconstructed. But this would require that the augmentation method is sophisticated enough for the original private data to be inaccessible (i.e. you should not be able to infer the original data from the augmented) while at the same time not reducing performance. Not an easy task by any means.

A potential application in the future for these types of methods could be for the detection of copyrighted material inside models. There have already been discussions in regard to the large language models and text-to-image models and their use of data that is copyrighted. If the training samples are actually memorized inside the model and effectively “combined” in various forms, what’s the difference between that and simply editing existing work yourself and publishing it?

Rewind.ai

There is an interesting new not-yet-released software that has been trending lately called Rewind (rewind.ai). The idea is that everything you do on your computer should be searchable, or as they put it: the search engine for your life. Here are the core features:

• Captures and saves text on your screen using optical character recognition (OCR)
• Captures and saves recordings and also makes them available for search using automated speech recognition
• Stores everything locally on your computer to protect privacy
• Uses compression to reduce the size of stored data, recordings are said to be compressed 3750 times
• Supposedly running it “feels virtually imperceptible”
• Only macOS currently

The ability to use functionality across apps is appealing. For instance, bookmarks are something many applications have, but because they are spread around different websites/apps it is difficult to keep track of everything. Website extensions/add-ons can be used for this purpose, but it’s not always a smooth experience and only works for websites.

The idea behind using OCR to function with any app appearing on the screen is clever. I would have expected it to be problematic performance-wise having it run constantly but supposedly it isn’t. The fact that everything is stored on your computer is key here, otherwise, an internet connection would be required, data would constantly be transferred and also privacy would be completely gone.

If successful, I would not be surprised if Apple acquired the company or built something similar themselves. Then the advantage would be that it could work both on your phone and computer, which is one problem that still remains. A search engine for your life cannot neglect your phone in this day and age.

It’s still only in the “early access”-phase, thus we are yet to see how well it works in practice.

That was all for this week, see you next week for another update!

Check out the next article in the series:

If you’re interested in reading more articles about data science or AI, check out my reading lists below:

If you’d like to get a Medium membership you can use my referral link if you wish. Have a nice day.