Speakers: Joe Liang and Xiaolong Yang
Transcript
Introductions
Xiaolong: It’s an honor to be in Connor’s class with my friend Joe and with all of you today. Today, we’re going to share our experiences and journey learning from a number of computer vision projects that we did over the past couple of months. And before cracking on, we’ll do a little self-intro. First is my friend Joe.
Joe: My name is Joe. I’m a first-year here at Harvard. I’m studying computer science and physics. And I met Xiaolong from my intro to computer science class, CS50. So we met early on in section, and then I saw something on his screen that was interesting. We talked about computer science for quite a long time, coding projects, and things like that. So this is the final project we worked on together and are trying to present.
Xiaolong: That’s right. And my name is Xiaolong. I was born in China. Growing up, I moved around a little bit. Did my high school in the UK and then college in Tokyo, Japan. And then now doing my grad studies at Harvard, working on applied statistics for political science. More specifically, I was interested in causal inference. I did a number of projects related to what Connor has been doing. Last semester, because I hadn’t formally taken any computer science classes before, I thought I would take one.
Luckily, I ran into Joe. We realized that we both use the knowledge organization software called Obsidian. That’s also why Joe reached out in one of the sections at first. We just decided to talk a little more about projects, and eventually we did this project together. And we’re really excited to share the whole journey we went on with you.
Connor: Let’s jump in. Thank you both for those introductions, and thank you for coming.
Project Demo
Joe: Yes, so the first thing we’d like to show is a demo of everything we’ve built, just to have you see what we’ve put up. What we built was a goat tracking system for a farm that Xiaolong’s family owns. So you have these goats passing down there. I guess we’ll talk more about it later, but in the demo at least you guys can get an idea of what it’s doing.
It’s just counting goats for the farmers. In the bottom left corner, you can see using a computer vision model to make sure you find the center point of each goat. And whenever they pass this gate, we’ll count it as one goat. Right there is a dashboard for all of this. On top is just a little simulation of some goats with some physics logic. All of it together is the final project we have, but the most important part is counting the goats—how significant that is and what we learned from it.
Xiaolong: That’s right. It is of Joe’s genius that we made this Boids simulation, where the idea is in a real goat setting, when the number of goats corresponds to the actual goats in the farm, you can actually draw the boundaries or fences where you can trap the goats and see how they react in certain behaviors. Because goats exhibit herd behavior, they follow one another. Oftentimes you want to optimize the route of where goats are going in a farm. We thought this is kind of a proof of concept that we wanted to provide to the farmers. It’s not the main work—the main work is here in the vision work—but it’s fun.
Connor: Can you talk a little bit about the motivation for goats in the first place? Was that a need on the farm that wasn’t being addressed? Was it expensive to do? Tell us more.
Motivation & Context
Xiaolong: Yeah, we will jump into that topic if that’s okay, because we have prepared a lot of slides for it. Before we get started on the actual thing, we want to give out a couple of caveats.
- Not Professional Experts: We’re not professional, serious computer science experts at this point. Everything we do is similar to what you would do when you see a project and then try to apply machine learning algorithms. We make mistakes, and one of our goals is to share them with you.
- Real-World Application Gap: We worked on a real problem, and it didn’t really work. Although you saw the technology working, applying it in the real world is a different story, and that’s one of the main things we want to elaborate on today.
- Value in Sharing: Despite all the seeming failures or mistakes, we still think everything is worth sharing.
The whole point of this talk is to motivate the thought that taking action on real-world problems is really the key. Everything that we will discuss henceforth will be in service of that message.
Where can you find problems? We read blogs, talk to people, and take classes to learn how to solve problems and how to know whether the problems we work on are worthwhile. From our limited experience so far, we think academic journals may offer cool solutions in theory, but in practice they often don’t work. Our intuition is not from academic journals to find important solutions or problems to work on.
It’s always good to talk to people who are close to you, who know a lot about the context and the problems that they face in their work and life. In this particular case, the problem arose from my own family, who know the challenges of running a cashmere farm. Talking with people you are close to understand the context of their problem will give you a much better edge in understanding and identifying key solutions.
The whole project started as a final project idea for the CS50 class that Joe and I took together. Before stumbling on this idea, we talked about creating social networking applications where we can connect our friends or remind us to say happy birthday to our friends. Over time we realized we wanted to work on important problems. I was reminded that China, and specifically my hometown in Inner Mongolia, has many goat farmers. These goat farmers face many difficulties in their careers.
Inner Mongolia is famous for two things: rare earth metals and these cashmere goats. Today’s story is about how those cashmere farmers face challenges in their life and how we saw those challenges and wanted to propose our own solutions.
The Goat Farming Industry
Xiaolong: Some general macro statistics just to give you a sense of the importance of the problem. Inner Mongolia alone supplies roughly 30% of the world’s cashmere. Over 3 million cashmere goats exist in the region. Some are wild, and some are raised by farmers. The share we wanted to help with is human-raised goats, which are facing severe breeding and disease problems.
Because Inner Mongolia alone supplies 30% of the world’s cashmere, the industry is very well capitalized. At the very bottom of the supply chain, you have the goats and the goat herders. One step up the supply chain, you have people who scavenge and collect the goat furs, put them into refineries, and turn them into raw yarns, which are what people use to make cashmere garments. Almost every single step of the supply chain is very well capitalized. International and domestic businesses invest heavily in building factories to run their industries as efficiently as possible.
But this is not the case at the very bottom end of the supply chain. Almost all farm-raised goats are raised in a highly segregated manner. By segmentation, we mean that the largest farm is usually a state-owned farm with a little over 10,000 goats. That is really a small number if you go to any other country and ask how many cattle, pigs, or chickens a big farm raises. It’s a very fragmented area.
These families, the goat herders, work tirelessly and laboriously every single day. Usually, it’s a senior couple. They wake up at 6:00 AM, go into the field, and actually live in the field in something called a Mongolian yurt—a movable tent that can change locations from spring to winter depending on where the goats graze. They are out in the field for the entire day, following their goats and making sure sufficient water is available. They take no breaks every single day, week, or year. It’s a very hard life.
On top of that, they don’t make much money. The hard truth is they don’t really have a business model; they just own these goats as commodities. They don’t have a stable stream of income and are unwilling to sell these goats to the market because once you sell 10 out of your 200 goats, you lose 5% of your net asset overnight. That’s really destabilizing. These seniors are still providing for their younger families with these goats, so it’s a high-stakes matter for them.
The worst problem of being a goat farmer is making sure breeding goes well. Breeding often goes wrong because these newborns are really fragile. You might go to sleep at 3:00 AM thinking the newborns will be okay, but wake up an hour later to find the baby is dead because of freezing cold air. It’s a really dire problem we wanted to help with.
In the field, we found a farm connected to my family. We conducted interviews to identify the bottleneck difficulties. We found that one of the main pain points is replacing on-the-ground labor. A large proportion of that labor is enumerating the right number of goats when they leave the pen and when they come back after grazing on the grassland.
If we can replace labor with computer vision technology to count the goats, then they don’t have to hire people, or they can be freed from this labor to do something else. Currently, they are doing it by eyeballing the goats every single day. We’ve tried to eyeball the goats using the security camera footage and it’s really hard. Without 20 years of experience, you just can’t count accurately.
Technical Approach & Challenges
Joe: So, Xiaolong mentioned this idea. Coming from my computer science background, I was like, “Oh, we can just insert a vision model to count the goats. Should be fine.” I’ve done a little bit of human tracking analysis before for sports. So I was like, okay, we just apply this model for goats and it should be okay.
The first model I looked at was SAM, the Segment Anything Model from Facebook. You run this and you can segment out things. The idea is just look at where it says “goats” and you can count the entire farm.
- Speed: Problem we ran into first, it was very slow. It takes a lot of computational power and time. This is not something that you can actually deploy on a real farm.
- Accuracy: Actually, it’s not too accurate either. The other big wall is that these models are not trained on this specific goat breed. These are special to Inner Mongolia. Computer vision models are good for humans because they are trained on data that includes photos of humans in different angles and actions. They don’t have much data on cashmere goats. When we tried to run these models, they wouldn’t run.
We tried to think systematically about how workers are counted. One method is to mix in different-colored goats—for every 50 white goats, you’d have one black goat. You just count the black goats and multiply by 50. But if you lose track of one black goat, you just lost 50 goats, which could mean losing a couple thousand dollars. Another way is to count only the horns, or to paint a distinct blue mark on the horns. But you could have blue that’s not on the goat, or something horn-shaped that gets confused. These goats are also very fluffy, which makes it even harder to track.
At the end, we went back to a vision model called YOLO (You Only Look Once). What this does is you can feed in your own data to fine-tune the model. We manually annotated many frames of what our specific goats look like. You stream in videos, draw bounding boxes, and teach the model what a goat looks like.
Another important thing for machine learning is it needs a benchmark or ground truth. We needed to understand exactly how many goats were actually in the video. We manually counted a half hour of video many times until we finally settled down on a number, a hundred something. That was the metric we used to evaluate the model.
Xiaolong: One little detail worth discussing is how humans count. How Joe and I counted could be seen as how people model problems differently. When Joe looked at the footage of goats going from bottom to top, if a goat reversed its direction, he would strike a horizontal line. My notes were mainly in numbers: 1, 2, 3, 4, 5, and 1, 1, 1, 2, 2, 3.
Once you open the gates, the goats swarm in. They don’t queue nicely like humans. Sometimes two or three squeeze in all at once, which is difficult for the model to pick up. By writing down numbers, you know how frequently goats come in as groups. At the end of the day, we can triage those ways of counting differently and use them to build a more comprehensive model.
Joe: We got the model to a decent accuracy. Xiaolong called the farm again and asked, “How accurate do you really want it to be?” The answer was 100%. They don’t want to miss a single goat. If you have 500 goats and the model says 499, you have to go look for it. We never actually achieved this, but we wanted to see how far we could go with deploying our prototype.
That’s when we ran into our second wall. Physical hardware deployment has much more issues than simulating it in your own environment. If the farm had good internet, you could stream the video to your local computer. But it’s Inner Mongolia, so wiring is difficult. If you bring a laptop to the goat farm, you face other issues like rain, latency, and generating battery power. All of these problems start to show up.
Xiaolong: And you wouldn’t believe how bad the stability of these models are. You would imagine after all the crazy hand-labeling, the model would perform beautifully at a 100% recognition rate. But whenever there’s rain, a sandstorm, or the camera pole is shaking, the recognition rate drops dramatically. The hand annotation overfits the model to a well-conditioned situation, whereas in real life, conditions change and destabilize performance.
Xiaolong: After sharing our journey, it’s pretty clear we didn’t transform the industry. But we did learn a number of things worth passing on:
- Work on Real Problems: Working on a real problem that you know will benefit others is one of the greatest motivations. Employing technology to save people time is impactful. Unlike classroom simulations, real-life problems are constrained (e.g., shipping deadlines for hardware), and it is your duty to optimize decision-making.
- Keep Things Simple: Don’t over-optimize in the beginning for parts you won’t use. We considered various complex models, only to find that you can’t deploy 80% of them in the field. Make things simple and don’t overcomplicate.
- Tailor the Technology: General technology is not often tailored to your specific problem. You have to be willing to put in the work to apply your own dataset and fine-tune it for your niche.
- Consider System Architecture: Automating the goat farming industry requires more than just deploying vision models. It means redesigning how factories run, combining software and hardware, and educating local people to use these technologies. It’s a system architecture problem.
- Work with Good People: Work with people you genuinely enjoy working with. You will run into problems, and if you enjoy each other’s company, it’s encouraging and easier to go through those hurdles.