Neuro Prosthetics and Brain Machine Interface

image40

About

Our research focuses on the following areas: 1) Brain-machine interface: The grand goal is to use the neural interface to control and sense the interactions with dexterous upper limb prosthesis. The signals used are EEG, Electrocorticogram, and neural action potentials. 2) Motor: We are developing algorithms for control of the dexterous prosthetic hand using advanced machine learning (e.g. sparse coding) and deep learning (temporal neural networks) for robust, position/load independent dexterous control.  3) Sensory: We design sensors that mimic human skin to provide a sense of touch, pressure, temperature and pain perception and provide electrocutaneous and vibration feedback information to the amputee. The sensory percepts are analyzed through both behavioral studies as well as brain signal (EEG) mapping and localizing the sources. 

Project Description

1. Brain-Machine Interface

The over-arching goal of upper limb prosthesis control is to provide intuitive, brain/mind controlled prosthesis. To reach this end, the central challenge is interpreting brain signals to derive the control signals from the brain. The key signals are Electroencephalogram (EEG), Electrocorticogram (ECoG), neural action potentials (spikes, local field potentials).  We have applied each of these signal techniques, each with its unique advantage and limitation, for demonstrating dexterous control in monkeys as well as humans.  The research pertained to neural signal analysis, network theory, decoding spikes/LFPs, and forming neural network and machine learning algorithms to interpret the patterns that decode for motor tasks.

image41

2. Motor

a) We are developing algorithms for control of the dexterous prosthetic hand.  After years of work on building brain-machine interface using EEG, ECoG, we are now back to the practical, affordable, non-surgical myoelectric EMG control. We use advanced machine learning (e.g. sparse coding) and deep learning (temporal neural networks) for robust, position/load independent dexterous control.  B) To facilitate linking motor functions to underlying motor systems of the limb, we are building a biomechanical and motor/sensory embodiment model.  C) To train an amputee, possibly at home, in the use of the dexterous hand, we employ augmented reality, HoloLens, and inertial unit technology – the ”Buzz-band”.

*replace with real image*

*replace with real image*

3. Sensory

An amputee also needs to feel and receive the sense of touch and corresponding embodiment of the sensory perception. Therefore, we design sensors that mimic human skin to provide a sense of touch, pressure, temperature and pain perception and provide electrocutaneous and vibration feedback information to the amputee. The se nsory percepts are analyzed through both behavioral studies as well as brain signal (EEG) mapping and localizing the sources.  Our goal is to demonstrate the sensory perception not only enhances the quality of sensory perception, but practically interact with objects in daily lives, and also to show improvements in the control of the prosthesis.

image42

Team

PI: Nitish Thakor, Professor

Students

Joseph Betthauser

Joseph Betthauser

Joseph Betthauser

Ph.D. student

Christopher Hunt

Joseph Betthauser

Joseph Betthauser

Ph.D. student

Teja Karri

Catherine Ding

Catherine Ding

Masters student

Catherine Ding

Catherine Ding

Catherine Ding

Masters student

Collaborators

Nathan Crone

Sridevi Sarma

Sridevi Sarma

 Professor

Neurology

Johns Hopkins Medical School

Sridevi Sarma

Sridevi Sarma

Sridevi Sarma

Associate Professor

Biomedical Engineering

Johns Hopkins University

Marc Schieber

Sridevi Sarma

Ralph Etienne-Cumming

Professor

University of Rochester

Ralph Etienne-Cumming

Ralph Etienne-Cumming

Ralph Etienne-Cumming

Professor

Electrical and Computer Engineering

Johns Hopkins University

Alcimar Soares

Ralph Etienne-Cumming

Andrei Dragomir

Professor, Director 

Biomedical Engineering Lab - FEELT - UFU - Brazil

Andrei Dragomir

Ralph Etienne-Cumming

Andrei Dragomir

SINAPSE

National University of Singapore

Anastasios Bezerianos

Anastasios Bezerianos

Anastasios Bezerianos

SINAPSE

National University of Singapore

Luke Osborn

Anastasios Bezerianos

Anastasios Bezerianos

Research Scientist

Johns Hopkins Applied Physics Labs

Matthew Fifer

Anastasios Bezerianos

Matthew Fifer

Research Scientist

Johns Hopkins  Applied Physics Labs

Rahul Kaliki

Rahul Kaliki

Matthew Fifer

CEO

Infinite Biomedical Technologies

George Levay

Rahul Kaliki

George Levay

Engineer

Infinite Biomedical Technologies

Alumni

PhD Students

Masters Students

Masters Students

 Sounyadipta Acharya

 Vikram Aggarwal

 Francesco Tenore

 Mohsen Mollazadeh

 Matthew Fifer

 Heather Benz

Masters Students

Masters Students

Masters Students

 Ani Chatterjee

 Michael Powell

 Martin Vilarino

 Bobby Beaulieu

 Matt Masters

Publications

Brain-Machine Interface

Fifer MS, Acharya S, Benz HL, Mollazadeh M, Crone NE, Thakor NV, “Toward Electrocorticographic control of a dexterous upper limb prosthesis: Building brain-machine interfaces,” IEEE Pulse, Vol. 3(1), pp. 38-42, Jan 2012. doi: 10.1109/MPUL.2011.2175636. Review. PMID: 22344950.


Thakor N. V., “Translating the brain-machine interface,” Sci. Transl. Med., Vol. 6 November, Vol. 5, Issue 210, p. 210ps17, 2013. DOI: 10.1126/scitranslmed.3007303.


Osborn L., Betthauer J. L., and Thakor, N.V., “Neuroprosthesis,” Wiley Encyclopedia of Electrical Engineering, J. G. Webster (Ed). 21 February 2019 https://doi.org/10.1002/047134608X.W1424.pub2.


Edelman B., Johnson N., Sohrabpour A., Tong S., Thakor N., He B., “Systems neuroengineering: Understanding and interacting with the brain,” Engineering, Vol. 1(3), pp. 292 -308, 2015.     DOI: 10.15302/J-ENG-2015078


Alam M., Rodrigues W., Bau Ngoc Pham B. N., and Thakor N. V., “Brain-machine interface facilitated neurorehabilitation via spinal stimulation after spinal cord injury: Recent progress and future perspectives,” Brain Res., Sep 1, 2016:1646:25-33. doi:


Liu R., Wang Y., Newman G. I., Ying S., Thakor N. V., “EEG classification with a sequential decision-making method in motor imagery BCI,” Int. J. Neural Systems, Vol. 27, No. 8, 2017. https://doi.org/10.1142/S0129065717500460.


Thomas T. M., Candrea D. N., Fifer M. S., McMullen M. S., Anderson W. S., Thakor N. V., Crone N. E., “Decoding native cortical representations for flexion and extension at upper limb joints using electrocorticography, “ IEEE Trans. Neural Systems Rehab Eng, Vol. 27(2), pp. 293-303, 2019. DOI: 10.1109/TNSRE.2019.2891362

Original Works

Acharya S, Tenore F, Aggarwal V, Etienne-Cummings R, Schieber MH, Thakor NV, “Decoding individuated finger movements using volume-constrained neuronal ensembles in the M1 hand area,” IEEE Trans Neural Syst Rehabil Eng, 16(1):15-23, 2008. http://www.ncbi.nlm.nih.gov/pubmed/18303801.


Aggarwal V, Acharya S Tenore F, Etienne-Cummings R, Schieber MH, Thakor NV, “Asynchronous Decoding of Dexterous Finger Movements using M1 Neurons,” IEEE Trans Neural Syst Rehab. Eng, 16(1):3-14, 2008. http://www.ncbi.nlm.nih.gov/pubmed/18303800 doi: 10.1109/TNSRE.2007.916289. Erratum in: IEEE Trans Neural Syst Rehabil Eng. 2008 Aug;16(4):421. PMID: 18303800.


Choi Y, Koenig MA, Jia X, and Thakor NV, "Quantifying time-varying multiunit neural activity using entropy based measures," IEEE Tran. Biomed. Eng, Vol. 57, pp. 2771 – 2777, 2010. http://www.ncbi.nlm.nih.gov/pubmed/20460201 doi:  10.1109/TBME.2010.2049266


Mollazadeh M., Greenwald E., Thakor N. V., Schieber M., Cauwenberghs G., “Wireless micro-ECoG recording in primates during reach-to-grasp movements,” IEEE BioCAS, pp. 237-240, 10 Nov 2011. PMID:23853286.


Mollazadeh M, Aggarwal V, Davidson AG, Law AJ, Thakor NV and Schieber MH, “Spatiotemporal variation of multi-modal neural activity in the primary motor cortex during dexterous reach-to-grasp movements,” J. Neurosci., Vol. 31(43), pp. 15531-15543, 2011. doi: 10.1523/JNEUROSCI.2999-11.2011. PMID: 22031899. 


Aggarwal V, Mollazadeh M, Davidson A, Schieber M, and Thakor NV, "State-Based decoding of hand and finger kinematics using neuronal ensemble and LFP activity during Dexterous Reach-to-Grasp Movements" J. Neurophysiol., Vol. 109(12), pp. 3067-3081, 2013. doi: 10.1152/jn.01038.2011.. doi: 10.1152/jn.01038.2011. PMID: 23536714


Benz HL, Zhang H, Bezerianos A, Acharya S, Crone NE, Zheng X, and Thakor NV, “Connectivity analysis as a novel approach to motor decoding for prosthesis control,” IEEE Trans. Neural Systems Rehab. Eng., 20(2), pp. 143-152, 2012.


Saxena S, Schieber MH, Thakor NV, and Sarma SV, “Aggregate input output models of neuronal populations,” IEEE Trans. Biomed. Eng., pp. 2030-2039, Jul 2012. PMID:  22552544


Kim H-N, Kim Y-H, Shin H-C, Aggarwal V, Schieber MH, and Thakor NV, “Neuron selection by relative importance for neural decoding of dexterous finger prosthesis control application,” Biomed. Signal Proc. Control,  Vol. 7 (2012) 632– 639, 2012.


McMullen D., Hotson G., Katyal K., Wester B., Fifer M., McGee T., Harris A., Johannes M., Vogelstein R.J., Ravitz A., Anderson W., Thakor N., Crone N., "Demonstration of a semi-autonomous hybrid brain-machine interface using human intracranial EEG, eye tracking, and computer vision to control a robotic upper limb prosthetic," IEEE Trans. Neural Systems Rehabilitation Eng., 2014. doi: 10.1109/TNSRE.2013.2294685. Epub 2013 Dec 12., PMID: 24760914.


Agarwal R., Thakor N., Sarma S. and Massaiquoi S., "PMv neuronal firing may be driven by a movement command trajectory within multidimensional Gaussian fields" J. Neurosci., Vol. 35(25): pp. 9508-9525; 24 June 2015.doi: 10.1523/JNEUROSCI.2643-14.2015.


Kahn K., Saxena S., Eskander E., Thakor N., Schieber M., Gale J. T., Averbeck B., Eden M. and Sarma S.,“Systematic approach to selecting task relevant neurons,” J. Neurosci. Methods., Vol. 245, , Pages 156–168, 30 April 2015.


Hotson G., McMullen D. P., Fifer M.S., Johannes M. S., Katyal K. D., Para M. P., Armiger R., Anderson W. S., Thakor N. V., Wester B. A., Crone N. E., “Individual finger control of the modular prosthetic limb using high-density electrocorticography in a human,” J Neural Eng. , 13(2):026017, 2016. doi: 10.1088/1741-2560/13/2/026017. Epub 2016 Feb 10.PMID: 26863276


Hotson G., Smith R., Rouse A. G., Schieber M., Thakor N., Wester B. "High precision neural decoding of complex movement trajectories using recursive Bayesian estimation with dynamic movement primitives", IEEE Robotics and Automation Letters (RA-L), pp. 676-683, Volume: 1, No. 2, July 2016, DOI: 10.1109/LRA.2016.2516590.


Hotson G., Fifer M. S., Acharya S., Benz H. L., Anderson W. S., Thakor N. V., Crone N. E.,”Coarse Electrocorticographic decoding of ipsilateral reach in patients with brain lesions, Plos One, 9(12):e115236. doi:10.1371/journal.pone.0115236.


Choi H., You K.-J., Thakor N. V., Schieber M.H., and Shin H.-C., “Single-finger neural basis information-based neural decoder (nBINDER) for multi-finger movements,” IEEE Trans. Biomed. Eng., vol. 26(12), pp. 2240-2248, 2018. 10.1109/TNSRE.2018.2875731

Motor

Powell MA and Thakor NV, “A training strategy for learning pattern recognition control for myoelectric prostheses,” J. Prosthetics Orthotics,” Vol. 25 , No. 1, pp. 30-41, 2013. PMID: 23459166.


Powell MA, Kaliki R, and Thakor NV, “User training for pattern recognition-based myoelectric prostheses: Improving phantom limb movement consistency and distinguishability,” PMID: 24122566. May;22(3):522-32, 2014. doi: 10.1109/TNSRE.2013.2279737. Epub 2013 Oct 7. PMID: 24122566


Fifer M.S., Hotson G., Wester B.A., McMullen D.P., Wang Y., Johannes M.S., Katyal K.D., Helder J.B., Para M.P., Vogelstein R.J., Anderson W.S., Thakor N.V., Crone N.E., "Simultaneous neural control of simple reaching and grasping with the modular prosthetic limb using intracranial EEG," IEEE Trans Neural Syst Rehabil Eng.Vol. 22(3), pp. 695-705, 2014. doi: 10.1109/TNSRE.2013.2286955. Epub 2013 Oct 24. PMID: 24235276.


Fifer M.S., Hotson G., Wester B.A., McMullen D.P., Wang Y., Johannes M.S., Katyal K.D., Helder J.B., Para M.P., Vogelstein R.J., Anderson W.S., Thakor N.V., Crone N.E., "Simultaneous neural control of simple reaching and grasping with the modular prosthetic limb using intracranial EEG," IEEE Trans Neural Syst Rehabil Eng.Vol. 22(3), pp. 695-705, 2014. doi: 10.1109/TNSRE.2013.2286955. Epub 2013 Oct 24. PMID: 24235276.


Ma J., Thakor N. V., and Fumitoshi M., “Hand and wrist movement control of myoelectric prosthesis based on synergy,” IEEE Trans. Human Machine System, 2014. doi: 10.1109/THMS.2014.2358634.


McMullen D., Hotson G., Katyal K., Wester B., Fifer M., McGee T., Harris A., Johannes M., Vogelstein R.J., Ravitz A., Anderson W., Thakor N., Crone N., "Demonstration of a semi-autonomous hybrid brain-machine interface using human intracranial EEG, eye tracking, and computer vision to control a robotic upper limb prosthetic," IEEE Trans. Neural Systems Rehabilitation Eng., 2014. doi: 10.1109/TNSRE.2013.2294685. Epub 2013 Dec 12., PMID: 24760914.


Kang X, Sarma SV, Santaniello S., Schieber M, Thakor N. V., “Task-independent cognitive state transition detection from cortical neurons during 3-D reach-to-grasp movements.,” IEEE Trans Neural Syst Rehabil Eng., Jul;23(4):676-82, 2015. doi: 10.1109/TNSRE.2015.2396495. Epub 2015 Jan 27.


Vilarino M., Moon J., Rogner Pool K., Varghese J., Ryan T., Thakor N., and Kaliki R., “Outcomes and perception of a conventional and alternative myoelectric control strategy: A study of experienced and new multi-articulating hand users,” J. Prosthetics Orthotics, Vol. 27(2), pp. 53–62, 2015. doi: 10.1097/JPO.0000000000000055.


Beaulieu R. J., Masters M. R., Betthauser J., Smith R. J., Kaliki R., Thakor N. V., and Soares A., “Multi-position training improves robustness of pattern recognition and reduces limb-position effect in prosthetic control,” JPO: J.Prosthetics Orthotics, Vol. 29 (2), pp. 54-62, 2017.


Betthauser J. L., Hunt C. L., Osborn L. E., Masters M. R., L´evay G., Kaliki R. R., and Thakor N. V., “Limb position tolerant pattern recognition for myoelectric prosthesis control with adaptive sparse representations from extreme learning,” IEEE Trans. Neural Systems Rehab. Eng., pp. 770 – 778, Vol.65, No. 4, 2017, DOI: 10.1109/TBME.2017.2719400.


Sumsky S. L., Schieber M., Thakor N. V., and Sarma S. "Decoding kinematics using task-independent movement-phase-specific encoding models," IEEE in Trans. Neural Systems Rehab. Eng., 2017. 10.1109/TNSRE.2017.2709756.

Sensory

Osborne L., Kaliki. R., Soares A., and Thakor N., “Neuromorphic event based detection for closed-loop tactile feedback control of upper limb prostheses" IEEE/ASME Trans. Haptics, Special Issue: Active Touch Sensing in Robots, Humans and Other Animals, Vol. 9 (2), pp. 196 – 206, 2016.


Lee WW, Kukreja SL, Thakor NV., “CONE: Convex-optimized-synaptic efficacies for temporally precise spike mapping,” IEEE Trans Neural Netw Learn Syst., 2016 Mar 24. [Epub ahead of print] PMID: 27046881.


Osborn L. E, Dragomir A. Betthauser J. L., Hunt C. L., Nguyen H. H., Kaliki R. R., and Thakor N. V., “Prosthesis with neuromorphic multilayered e-dermis perceives touch and pain,” Sci. Robotics, 3, eaat3818 (2018) 20 June 2018.


Low J. H., Lee W.W., Khin P.M., Thakor N. V., Kukreja S.L., Ren H.L., Yeow C.H., “Hybrid tele-manipulation system using a sensorized 3-D-printed soft robotic gripper and a soft fabric-based haptic glove,” IEEE Robotics and Automation Letters, Vol. 2 (2), pp. 880-887, 2017.