We propose a human-in-the-loop learning architecture which addresses the question of how learning can be achieved for tightly coupled physical interactions between the learning agent and a human partner. In recent years, the application domains of humanoid robots continue to expand, moving deeper into the realm of everyday life. Thus recent robotic developments are increasingly targeted at domestic environments and assistive tasks, in which human-robot interaction is indispensable. In order for humans and robots to engage in direct physical interaction, we employ a flexible joint humanoid robot driven by pneumatic actuators. This paper presents an example for such human in-the-loop learning scenarios and proposes a computationally efficient learning algorithm for this purpose. The efficiency of this method is evaluated in an experiment, where human care givers help an android robot to stand up.