Artificial Democracy

I am studying machine reinforcement learning for my master's thesis. Today I stumbled across a concept called "ensembles." There are many different algorithms to which can be used to make a computer learn how to do something that may be difficult to explicitly describe. For example, how do you make a helicopter hover?

You could study the motion of the helicopter and formulate precise equations to control the blades. Reinforcement learning is interesting because you can teach the helicopter that it has blades and it can spin them quickly or slowly; this way or that way. Then you teach the helicopter that falling or shaking a lot are bad. If it does either of these, you punish the helicopter. Over time, the helicopter can learn how to hover because it is trying to get the most "reward" that it can by its actions. There are several algorithms that are suitable to different types of problems. We might choose one of these algorithms and program the helicopter to "learn" by that method.

The approach that I learned about today is like an artificial democracy. Instead of choosing one algorithm under the rules of which the helicopter must learn, you choose several algorithms at the same time. Each algorithm will come up with its own policy of how to fly the helicopter to maximize its own interpretation of the "reward."

Next, the helicopter listens to all of the algorithms at once, either by taking some average of biasing towards some rather than others. It would be as if the helicopter had a group of policy advisers who would provide suggestions. I read a dissertation today which argued that ensembles of policies might actually be more robust than individual policies for certain problems. Perhaps ensembles could work well on problems which are "in-between" areas of specialization for the member algorithms or problems for which very little information is known.

This idea is very intuitive and seems very related to how living systems and human-made systems work (like democratic governments or ant colonies).

The Girlfriend Problem

Anyone who has ever had a girlfriend knows about the problem of feedback attribution.

Perhaps you call your girlfriend to see if she wants to go on a date. When she answers, she seems upset about something, but refuses to tell you what is specifically upsetting her. A man somewhat experienced with women can (sometimes) figure out the reason why she is upset. He can think of what he has done in the recent past and make consider some of the things that perhaps he shouldn't have done. He is able to select a small group behaviors or actions from the totality of his existence that probably made his girlfriend upset and use this knowledge to determine how to apologize properly.

How is it that a man is able to figure out why a woman is mad when she refuses to tell him? Can a robot learn to do this too?

Perhaps Fred knows that his girlfriend, Sarah, usually finds out about his mischievous behavior after 14 days with a variance of 3 days. When trying to apologize to Sarah, he can assume that the she is upset because of an action approximately 14 days in the past. Fred promises not to repeat any of his actions that took place between 11 and 17 days ago, hoping that she will then forgive him.

Should can spread Sarah's disgust over his past actions in many ways to try and determine what it was that he did wrong. He can use any of the stochastic distributions to associate Sarah's frustration with the appropriate actions. Fred might try to take into account the frequency of his actions and Sarah's usual disposition to produce a better guess.

For example, Fred plays golf with his friends each week. Normally, Sarah is quite content during these weeks. Fred decides that because of how often he golfs, Sarah is probably mad for a different reason. he notices that 12 days ago was the only time he can remember when one of his golfing excursions overlapped with Sarah's birthday. It is very unusual for these two events to overlap, so Fred decides that there is a high probability that Sarah is upset about that and decides that in the future, he will either try to reschedule his golfing trip or Sarah's birthday.

An intelligent machine might function in a similar way. Consider a machine that is designed to play diplomacy. As the game progresses, the machine chooses actions that benefit its own nation. However, the machine is attacked by Russia, who had previously been very peaceful. The machine must decide why this betrayal occurred. Did one of the machine's past moves offend Russia? Did a past move hit one of Russia's allies?

The difficulty of the problem is that a machine does not know when its actions will be judged. An intelligent machine must be able to learn from the judgments (good or bad) that are bought upon it. Humans are able to ask themselves, "Why did this happen to me?" We are pretty good at determining what we did in the past to deserve our punishment or reward. From that knowledge, we learn whether to do it more or less.

The Biomimicry Taxonomy

I've become fascinated with using biomimcry (copying nature) to try and address human problems. This approach blends some of the most interesting aspects of science and mathematics. The goal is to learn from the inventions which have evolved in nature because they must already obey all physical laws (including the ones we haven't yet discovered).

The reason I'm writing today is because I came across a fascinating website in my internet wanderings. It's called Ask Nature. The idea of the website is to create a free, open source database of natural designs.

There is an image available on the website called the biomimicry taxonomy and it provides a visual for the different functions that organisms or natural processes perform.

The idea is this: if you have a problem, you usually have a main question. How can I store energy in the electrical grid? How can I make more food from the same plot of land? How can I control hazardous wastes? The website challenges you to ask these questions in a different way: How does nature store energy? How does nature produce food? How does nature recycle wastes?

The beauty of nature is that the inventions which have evolved are typically ecologically balanced. Sustainability is essential because organisms that are not sustainable become extinct. Perhaps we too can become sustainable if we learn from all of the other species which have been around far longer than us.

For example, when I navigate to energy storage techniques, there is an entry about how some bacteria use a kind of thermoplastic polyester to store carbon and energy. Learning more about these types of bacteria might inspire some inventions in biodegradable plastic.

Sim City in Real Life

Imagine a website where you could view a city like you do in Google Earth, but the streets were coloured based on traffic congestion like in Sim City.

I'm reading a book called Intelligent Infrastructure. Real-time traffic visualizations like this might be possible with the algorithms the book discusses. The math seems pretty complicated so far, so I won't pretend to understand the details yet, but the hand-waving notion of it is intriguing.

Traffic Congestion by Colour

The sensors gather data and the neural networks "learn" what "good" and "bad" traffic look like. They could have several increments and colour the road based on these ratings. The user could see the colours in real time (maybe updated every 30 seconds or so). It could be a neat tool for commuters and for city governments alike. It could even be used to detect accidents and automatically send emergency crews. Imagine a smarter city that orders services for its own roads.

Happiness -> Success NOT Success -> Happiness

This blog post was inspired by a TED video (as have been many others) called The happy secret to better work. The presenter is Shawn Achor and he studies positive psychology. I've read a bit about positive psychology as a set of tools for improving one's life in many regards.

I won't summarize the TED talk -- it would be like trying to summarize a painting -- but I do want to note the conclusions of positive psychology that he referred to in his talk.

He had a slide entitled "Small Changes Ripple Outward." It lists five examples of practices which can lead to lasting positive change in a person. I have implemented these strategies in my life and I can attest that they work phenomenally. They are:

1. 3 gratitudes (Emmons & McCullough, 2003)
2. Journaling (Slatcher & Pennebaker, 2006)
3. Exercise (Babyak et al., 2000)
4. Meditation (Dweck, 2007)
5. Random acts of kindness (Lyubomirsky, 2005)

I included the authors in case you are interested in reading more about each tool like I am.

What I got from the lecture is that you should make life choices based on whether you will be happy with the results of your decision. For example, if you are choosing a career, you should choose it based on whether you will experience flow in your daily activities.

Flow is a concept with which I first became familiar during summer 2011, when I was doing a lot of introspective thinking. I would describe it as being wholly engaged and mentally involved in whatever it is you are doing. It is living entirely in the present either for just a moment or for an extended period.

Thinking about it, I've experienced this mental state and I'm sure that you have too. For me, it happens when working on an intricate puzzle or problem. My mind is devoid of everything other than the task at hand. I've experienced it a lot lately when I've been working on my 4th year project in the robotics lab at school.

Perhaps you have experienced it too when spending quality face-to-face time with your wife/husband/boyfriend/girlfriend or when playing a remarkably engaging video game. Perhaps it was when you went for a bike ride or a hike and took in the beauty of the world around you without at all considering where in this world you were expected to be next. To an athlete, it might be called "being in the zone" or for a gamer it might be "pwning."

The term was created by Csíkszentmihályi, a researcher in this area. (Don't ask me how to pronounce it) It came from people describing themselves as being carried by a current of water while experiencing the concept of flow. There's also a TED video about flow. In the wikipedia article about it, there are ten factors which accompany the experience:

1. Clear goals (expectations and rules are discernible and goals are attainable and align appropriately with one's skill set and abilities). Moreover, the challenge level and skill level should both be high.
2. Concentrating, a high degree of concentration on a limited field of attention (a person engaged in the activity will have the opportunity to focus and to delve deeply into it).
3. A loss of the feeling of self-consciousness, the merging of action and awareness.
4. Distorted sense of time, one's subjective experience of time is altered.
5. Direct and immediate feedback (successes and failures in the course of the activity are apparent, so that behavior can be adjusted as needed).
6. Balance between ability level and challenge (the activity is neither too easy nor too difficult).
7. A sense of personal control over the situation or activity.
8. The activity is intrinsically rewarding, so there is an effortlessness of action.
9. A lack of awareness of bodily needs (to the extent that one can reach a point of great hunger or fatigue without realizing it)
10. Absorption into the activity, narrowing of the focus of awareness down to the activity itself, action awareness merging.

Read number eight again. And once more. Work becomes effortless? That sounds good to me. It's for this reason that I think that flow should be an important consideration in all life decisions. You must ask yourself, "Will I be able to have flow if I do this?" It encompasses everything that is needed to be a joyous, creative person with a zest for living. After you've asked and answered this question, then you can ask questions like, "How much will I make? How famous will I become? What will other people think of me?" But if you have your first answer, the other answers won't matter much.

Shawn Achor argues that happiness would ultimately lead to success. In my experience, he couldn't be more correct. Looking back, the only times I've failed at something in life were caused by improper consideration of flow. I've held jobs that I hated and, upon reflection, I never experienced flow in any of them. I've held, (and currently hold), jobs that I loved and each of them allowed me to experience flow at least a large majority of the time.

Success does not lead to happiness. It's the other way around. Why don't they teach us this in school when we're kids? I can't argue with the data.