NotionNext BLOG

Simulation of Cochlea-Implants

Fri, 24 Mar 2023 00:00:00 GMT

Background

The mechanics of our ear convert sound into vibrations of the basilar membrane. The way this works:

Figure 1:The basilar membrane, located in the cochlear is small and tight at one end, leading to resonances at high frequencies.

Figure 2: And it is wide and loose at the other end, leading to resonances at low frequencies.

This means that our ear automatically performs approximately a Fourier Transformation on the incoming sound, separating the high frequency compents from the low frequency components. This feature is used in the extremely successful design of cochlea implants (CIs): There sound is separated into approximately 20 frequency bins, and 20 electrodes stimulate the corresponding auditory nerve cells:

Figure 3: Artist's drawing of an inserted cochlear implant.

Limitation of CI-Electrodes

Figure 4: The number of electrodes for CIs is limited mainly by (1) current spread (green), and by (2) minimum electrode size required to avoid large current densities (red).

Possible Solutions: Time Domain or Frequency Domain

Figure 5: For each time window, the stimulation intensity of each electrode can be worked out either in the Time Domain or in the Frequency Domain.

(a) Time Domain: Linear Filters - Gamma Tones

In fact, the basilar membrane does not behave like a Fourier Transform. To a first approximation, Gamma tones describe the frequency response of the basilar membrane quite well. They can be well -- and very efficiently -- be approximated by IIR-Filters. Note the "Multi-resolution behavior" of the gamma tones: high frequencies decay at a much faster (time)scale than the lower frequencies:

Figure 6: Effect of a sound impulse at four different locations on the basilar membrane.

(b) Frequency Domain: Powerspectrum

One way to characterize the frequency dependence of a signal is to use a Fourier Transformation (FFT).

The equation for the frequency components of an FFT is , where "N" is the number of data-points, "Ts" the sampling-interval, and "n" the running index (1:N)

Frequencies on the basilar membrane are arranged approximately logarithmically. Take this fact into consideration.

Exercise Description: Simulation of Cochlea-Implants

Write a function that simulates cochlea implants:

Your function should allow the user to interactively (=graphically) select an audio-file.

Your function should produce as output a WAV-file, containing the CI-simulation corresponding to the selected audio-file.

The simulated CI should have approximately the following specifications:

= 200 Hz
= 500 - 5000 Hz
numElectrodes = 20
StepSize = 5 - 20 ms

Provide your program with an option to try out the n-out-of-m strategy used in real cochlear implants: at any given time, only those n (typically 6) of all m available (typically around 20) electrodes are activated, which have the largest stimulation.

Possible Program Structure:

Select file

Read data (mp3, wav)

Set parametersnumber of electrodesfrequency rangeWindow size (sec)[Window type][sample rate]

[CalculateWindow size (points)Electrode locations]

Allocate memory

For [all data]Get dataCalculate stimulation strengthSet/write output values

Play result

Figure 7: Possible workflow.

Implementation

In the implementation, we strictly follow the principles which have been mentioned before.

Read in the data

First, we need to read in the original sound. In order to process the data more easily, we take the average of multiple channels.


# Read data.
sound = sounds.Sound()
if len(sound.data.shape) > 1:
    # Average on multiple channels.
    sound.data = np.mean(sound.data, axis=1)

Parameter settings

We set the parameters according to the suggestions in the document of requirements. To make the parameter setting more modularized, we set the parameters in a YAML file and create a class to structurally contain the parameters. The parameter settings are:


ex1:
  m_electrodes: 22  # Total number of electrodes
  n_channels: 9  # Number of activated channels
  lo_freq: 200  # Hz
  hi_freq: 5000  # Hz
  period_window: 6.0e-3  # sec
  step_window: 5.0e-4  # sec
  type_window: 'moore'  # Filter window type

What’s more, we also create a graphical interface to allow the users change the parameters interactively.


# Set parameters. Load the parameters from a YAML script.
path_crt = os.getcwd()
Params = Params_audio(path_options=os.path.join(path_crt, 'options.yaml'))

# Create a graphical interface to visualize and change the parameter settings.
Params = gui_params(Params=Params)

# Calculate other parameters.
size_window = round((Params.period_window/sound.duration)*sound.totalSamples)  # window size (points)
size_step = round((Params.step_window/sound.duration)*sound.totalSamples)  # step size (points)

Apply filters to the original audio

In this process, we set up a series of IIR filter banks and then apply them to the original audio. Next, we simulate the n-out-of-m strategy. The principle is, during the proccessing time, only n (which have the strongest stimulations) out of m electrodes (all available electrodes) will be activated. In this way, we need to calculate the signal intensities of each electrode and only select a part of them to activate. Then, we need to calculate the corresponding amplitudes for the subsequent signal reconstruction process.


# Set up IIR filter banks.
(forward, feedback, fcs, ERB, B) = GTS.GT_coefficients(
    fs=sound.rate, n_channels=Params.m_electrodes, 
    lo_freq=Params.lo_freq, hi_freq=Params.hi_freq, method=Params.type_window
    )
    
# Apply filter banks on the whole audio.
data_filtered = GTS.GT_apply(sound.data, forward, feedback)

# Simulate the n_out_of_m process.
amp_n_out_m = n_out_m(
    data_audio=data_filtered, size_window=size_window, 
    size_step=size_step, Params=Params
    )

Audio reconstruction

After the process of n-out-of-m strategy, we get the amplitudes of each frequency band during a time period. In order to reconstruct the output audio, we can make use of sine/cosine functions with these specific amplitudes. After generating sine/cosine functions, add them linearly to get final result.


# Audio reconstruction.
y_output = reconstruction(
    amps=amp_n_out_m, size_step=size_step, totalSamples=sound.totalSamples, 
    duration=sound.duration, fcs=fcs
    )
# Write to output file
sound_output = sounds.Sound(inData=y_output, inRate=sound.rate)
sound_output.write_wav(full_out_file=os.path.join(path_crt, 'output.wav'))

Demo

Here we present several examples of the simulation results. We use two original sounds provided by the requirement document and compare them with the reconstruction outputs. The parameter settings are the same with as before.

tiger.wav

Original:

Simulation:

scales.wav

Original:

Simulation:

harmony.wav

Original:

Simulation:

According to the results of comparison, we can conclude that our simulation results can reconstruct the original sounds with a relatively low quality. One of the reasons is that we apply the n-out-of-m stategy during the simulation process. So only several frequency bands are activated.

In the interest of brevity, we only present the main structure of the simulation process. The source code can be found in my GitHub.

Optimal Facial Regions for RPPG Algorithms in HR Estimation

Tue, 02 May 2023 00:00:00 GMT

Brief Information

It is a research project (2023/05/16-2023/10/02) conducted by Shuo Li in the BMHT Lab. This project was continuously supervised by Dr. Moe Elgendi and the project advisor was Prof. Dr. Carlo Menon. The main background of this project is using rPPG algorithms for heart rate (HR) estimation. Our main goal is to compare the performance of different facial ROIs and try to reach out some consistent conclusions. More details can be in the following links:

Written Thesis:

ProjectThesis_shuoli.pdf

Slides (for presentation):

OptimalROI_Pre_shuoli.pdf

Literature Review (Under Review in Communication Medicine)

OptimalROI_Review_shuoli.pdf

Optimal ROIs for different subject’s movements (Under Preparation):

OptimalROI_Motion_shuoli.pdf

Source Code:

The source code of this research project has been uploaded to my GitHub. 🧐

Simulation of a Vestibular Implant

Tue, 02 May 2023 00:00:00 GMT

Background

Figure 1: A diagram of the vestibular system.

The vestibular system is a sensory system that is critically important in humans for gaze and image stability as well as postural control. Patients with complete bilateral vestibular loss are severely disabled and experience a poor quality of life. There are very few effective treatment options for patients with no vestibular function. Over the last 10 years, rapid progress has been made in developing artificial 'vestibular implants' or 'prostheses', based on cochlear implant technology. As of 2019, 19 patients worldwide have received vestibular implants and the results are encouraging. Vestibular implants are now becoming part of an increasing effort to develop artificial, bionic sensory systems.

Exercise Description: Simulation of a Vestibular Implant

Data

The files Walking_01.txtand Walking_02.txtcontain the linear accelerations and angular velocities recorded when walking in a figure-of-eight, around a table and a chair.

Methods

For measuring the movement a sensor by XSens was used, with the orientation on the head as indicated in the figure below.

At the start of the recordings, the subject was stationary, and the head was in an orientation with Reid's line 15 deg nose up.The sensor orientation on the head is such that at t=0, the "shortest rotation that aligns the y-axis of the sensor with gravity" brings the sensor into such an orientation that the (x/ -z / y) axes of the sensor aligns with the space-fixed (x/y/z) axes. This requirement uniquely determines the sensor-orientation at t=0.

Data were sampled at 50 Hz.

The acceleration data are given in m/s2, the angular velocity data in rad/s.

Figure 2: A diagram of the walking process and the settings of coordinate systems.

Task 1: Simulate the vestibular neural response

Simulate the neural vestibular responses during walking, using only the 3D-linear-acceleration and the 3D-angular-velocity from the file Walking_02.txt:

Right horizontal semicribular canal

Calculate the maximum cupular displacements (positive and negative). These two values should be written into the text file CupularDisplacement.txt.

Otolith haircell

Assume that an otolith hair-cell has at t=0 the on-direction=[0 1 0] in space fixed coordinates (i.e. pointing to the left as seen from the subject). Calculate the minimum and maximum acceleration along this direction, in m/s^2 and write this value to the text file MaxAcceleration.txt.

Task 2: Calculate the “nose-direction” during the movement

Calculate the orientation of the “Nose”-vector (as indicated in the figure) at the end of walking the loop (for Walking_02.txt),

based on the angular velocity readings,

and assuming that at t=0: nose=[1,0,0].

Input data

The folder MovementData contains the following files:

Walking_01.txt, Walking_02.txt: Linear acceleration and angular velocity recordings from human subject while walking around.

SCC_Humans.m: Orientations of the human semicircular canals, with respect to "Reid's plane". (Reid's plane is the plane defined by the lower rim of the orbia, and the center of the external auditory meatus. In "English": the bottom of the eyes, and the middle of the ears.)

Tips

Parameters

The radius of the human semicircular canals is 3.2 mm.

For the given input, the magnitude of the cupular displacement is about +/- 0.12 mm.

For the given input, the magnitude of the maximum acceleration along this axis is between -5.63 m/s^2 and +6.87 m/s^2

Implementation

In the implementation, we divide the simulation into two parts: Task 1 and Task 2.

Read in the data

Before implementing the simulations of task 1 and task 2, we need to read in the movement data.


# Load sensor data
dir_data = os.getcwd()
data_sensor = XSens(os.path.join(dir_data, 'MovementData', 'Walking_02.txt'))

Task 1

What’s more, we also create a graphical interface to allow the users change the parameters interactively.

In task 1, we need to finish two parts of simulations:

Calculate the maximum cupular displacements.

Calculate the minimum and maximum acceleration along this direction.

Data decomposition

In this process, the pre-loaded data is decomposed and some parameters are calculated.


## Data decomposition
acc = data.acc  #  Acceleration
omega = data.omega  #  Angular velocity
rate = data.rate  #  Sample rate
N = data.totalSamples  #  Number of samples

## Parameter calculation
vec_g_hc = np.array([0, 0, -0.981])  #  Gravity in Zurich. Head coordinates.
vec_g_approx = np.dot(sk.rotmat.R(axis='x', angle=90), vec_g_hc)
vec_ref = acc[0, :]  #  Take acc(t=0) as the reference vector.

Calculate the cupular displacement

With the input data and parameters, we can compute the cupular displacement. First, we need to adjust the original angular velocity data from the sensor coordinate system into the head coordinate system. Then, we compute the orientation of the SCCs of humans’ right canal and multiply it with the adjusted angular velocities. The result is the stimulation. Finally, we can calculate the cupular displacement using the stimulus, sampling rate, number of total samples and the radius of SCCs.


## Displacement of the cupula
# Adjust the angular velocities.
omega_adjusted = alignment(data=omega, vec_ref=vec_ref, vec_g=vec_g_approx)
# Orientation of the SCCs in humans.
canal_left, canal_right = sccs_humans()
# Calculate the stimulation using the dot product.
stimulus = np.dot(omega_adjusted, canal_right[0, :])
# Calculate deflection
deflection_cupula = cal_deflection(stimulus, rate, N)
# Calculate displacement
radius_canal = 3.2  #  Radius of the SCC
displacement_cupula = deflection_cupula * radius_canal

The acceleration data can be calculated by adjusting the original acceleration data into the head coordinate system. The otolith acceleration is in the on-direction [0,1,0].


## Acceleration along the on-direction [0 1 0] in the Head coordinates.
# Adjust the acceleration.
acc_adjusted = alignment(data=acc, vec_ref=vec_ref, vec_g=vec_g_approx)
# Preserve the component in the on-direction [0, 1, 0].
acc_otolith = acc_adjusted[:, 1]

Preserve the results

We save the results of task 1 locally. For simplicity, we only preserve the maximum and minimum data.


## Write the values in text files.
# Cupular Displacement
path_cupular = os.path.join(dir_crt, 'output/CupularDisplacement.txt')
with open(path_cupular, 'w', encoding='utf-8') as f:
    f.writelines(f'Maximum cupular displacement (positive): \
                 {np.max(displacement_cupula)} mm\n')
    f.writelines(f'Maximum cupular displacement (negative): \
                 {np.min(displacement_cupula)} mm\n')
print('The maximum cupular displacements have been saved to: ')
print(path_cupular)
# Acceleration
path_acc = os.path.join(dir_crt, 'output/MaxAcceleration.txt')
with open(path_acc, 'w', encoding='utf-8') as f:
    f.write(f'Maximum acceleration along the direction of the otolith hair cell: \
            {np.max(acc_otolith)} m/s^2\n')
    f.write(f'Minimum acceleration along the direction of the otolith hair cell: \
            {np.min(acc_otolith)} m/s^2\n')
print('The maximum and minimum acceleration have been saved to: ')
print(path_acc)

Task 2

In task 2, we need to finish two parts of simulations:

Calculate the “nose-direction” during the movement.

Visualize the quaternions to describe the head orientation.

Caculate the nose orientation

In order to calculate the nose orientation, we need to get the head orienation. In the head coordinate system, we can rotate the angular velocities according to the relative pose of the Reid’s line and use quaternions to compute the head orientation. The nose orientation can then be computed based on the head orientation and the fact that “at t=0: nose=[1,0,0]”.


# Head orientation
orientation_head = cal_head_orientation(omega_adjusted)
    
# Nose orientation
orientation_nose = []
for i in range(orientation_head.shape[0]):
    R_tmp = sk.quat.convert(orientation_head[i, :], to='rotmat')
    # t=0: nose=[1 0 0]
    orientation_nose.append(np.matmul(R_tmp, np.array([1,0,0])))
orientation_nose = np.array(orientation_nose)

# Write the values in text files.
path_nose_dir = os.path.join(dir_crt, 'output/Nose_end.txt')
with open(path_nose_dir, 'w', encoding='utf-8') as f:
    f.write(f'The orientation if the "Nose"-vector at the end of the walking loop: \
            {orientation_nose[-1]} m/s^2\n')
print('The nose vector at the end of the walking loop has been saved to:')
print(path_nose_dir)

Visualize the head orientation

The head orientation is represented in the form of quaternions. We can visualize the head orientation at each moment using a curve graph.


# Visualization of the head orientation
plt.plot(np.linspace(0, 20, orientation_head.shape[0]), orientation_head[:, 1], color='blue')
plt.plot(np.linspace(0, 20, orientation_head.shape[0]), orientation_head[:, 2], color='green')
plt.plot(np.linspace(0, 20, orientation_head.shape[0]), orientation_head[:, 3], color='red')
plt.grid(linestyle='--')
plt.xlim([0, 20])
plt.ylim([-0.4, 1.0])
plt.title('Head Orientation')
path_img = os.path.join(dir_crt, 'output', 'head_orientation.PNG')
plt.savefig(path_img)
print('The visualization result of the head orientation has been saved to:')
print(path_img)

3D visualization of the nose orientation

After computing the nose orientation, we visualize the 3D process of the movement. We save the simulation result as a GIF file. In the implementation, we write a seperate function to reconstruct the 3D process.


def visualization_nose(orientation_nose, dir_crt):
    """3D visualization of the nose orientation.

    Parameters
    ----------
    orientation_nose: Nose orientations during the movements.
    dir_crt: Directory of the current folder.

    Returns
    -------
    
    """

    fig = plt.figure()
    # Initialize the 3D coordinates.
    ax = fig.add_subplot(projection='3d')
    # Visualize the nose vectors dynamically.
    x = np.linspace(-1, 1, 5)
    y = np.linspace(-1, 1, 5)
    X, Y = np.meshgrid(x, y)
    for i in range(len(orientation_nose)):
        # Progress meter
        PSG.one_line_progress_meter(
            'Video frame generating.',
            i+1,
            len(orientation_nose),
            '',
            'Generating frames of the nose orientation...'
        )
        # Clear the current figure.
        plt.cla()
        # 3D surfaces for better visualization
        ax.plot_surface(X, Y, Z=X*0, color='g', alpha=0.2)
        ax.plot_surface(X, Y=X*0, Z=Y, color='y', alpha=0.2)
        ax.plot_surface(X=X*0, Y=Y, Z=X, color='r', alpha=0.2)
        # Plot a 3D arrow (Nose).
        vec_tmp = orientation_nose[i]
        ax.quiver(0, 0, 0, 
                  vec_tmp[0], vec_tmp[1], vec_tmp[2], 
                  arrow_length_ratio=0.2, color='black', normalize=True)
        # Plot settings
        ax.set_xlabel('x')
        ax.set_ylabel('y')
        ax.set_zlabel('z')
        ax.set_xlim(-1.0, 1.0)
        ax.set_ylim(-1.0, 1.0)
        ax.set_zlim(-1.0, 1.0)
        plt.title('myNose ' + str(i) + '/' + str(len(orientation_nose)))
        path_tmp = os.path.join(dir_crt, 'output', '3D_tmp', str(i)+'.PNG')
        plt.savefig(fname=path_tmp)
    # imageio
    gif_images = []
    for i in range(len(orientation_nose)):
        # Progress meter
        PSG.one_line_progress_meter(
            'GIF generating.',
            i+1,
            len(orientation_nose),
            '',
            'Generating the GIF of the nose orientation...'
        )
        # GIF images
        path_tmp = os.path.join(dir_crt, 'output', '3D_tmp', str(i)+'.PNG')
        gif_images.append(imageio.imread(path_tmp))
    
    # Save the output
    path_save = os.path.join(dir_crt, 'output', 'myNose.gif')
    imageio.mimsave(path_save, gif_images, fps=50)
    print('The GIF which shows the nose orientation has been saved to:')
    print(path_save)

Demo

Here we present some results of this simulation task. The running time often takes 1-2 minutes (most of the time is used by 3D visualization).

Figure 3: The head orientation during the moving process.

Figure 4: 3D visualization of the nose orientation.

The complete source code can be found in my GitHub.

Simulation of a Retinal/Visual Implant

Tue, 02 May 2023 00:00:00 GMT

Background

The idea of a “visual prothesis” is not as far fetched as it might seem at first, and a large number of research groups are working on different approaches. A recent overview can be found in Retinal Prothesis (Bloch, 2019). The exercise description part was created by Prof. Dr. Thomas Haslwanter.

Exercise Description: Simulation of a Retinal/Visual Implant

Data

All the Files for this exercise are bundled in “Ex_Visual.zip”

In addition, you can use the following files:

Typical standard test images that are often used in image processing (e.g. lena, mandrill, etc.) can also be found at the Waterloo BragZone.

You can also use one of the following:

Figure 1: TheDoor.jpg (146 kB)

Figure 2: eye.bmp (434 kB)Figure 2: eye.bmp (434 kB)

Figure 3: lena.tif (769 kB)

Hans van Hateren hosts a website with natural images that people often use for training receptive fields, etc.

General Requirements

For this exercise you should design a "visual prosthesis": Write a Pyhon program which

Takes a given input image, or - if none is provided - lets you interactively select an input image

In this image, lets you interactively select a fixation point ("ginput")

Calculates the activity in the retinal ganglion cells, and shows the corresponding activity, and

Calculates and shows the activity in the primary visual cortex, and

Save the images to an out-file.

Retinal Ganglion Cells

Assume that

the display has a resolution (for those 30 cm) of 1400 pixels,
and is viewed at a distance of 60 centimeter (see Figure below),
and that the radius of the eye is typically 1.25 cm.

This lets you convert pixel location to retinal location.

We know that the retinal ganglion cells respond best to a "center-surround" stimulus: they show the maximum response when the center is bright and the surrounding dark ("center-on cells"), or vice versa ("center-off cells"). This behavior can be simulated with a "Difference of Gaussians" (DOG)-filter. For this exercise, simulate only "center-on" responses. The figure below shows a section through the receptive field of a typical ganglion cell. The receptive field of such a cell can be simulated with a "difference-of-Gaussians" (DOG)-filter with the following ratio for the standard deviations of the two Gaussians:

From the figure below we see that the sidelength of the receptive field should be about

so that the response can go back approximately to zero at the edges (which happens at about ).

The receptive field size increases approximately linearly with distance from the fovea. For this exercise we simulate only magnocellular cells, the receptive field of which have a receptive field size of approximately

The parameters are also described in the wikibook on Sensory Systems, and in the article Computational models of early human vision, which contains the following image for M- and P-cells:

Note:Take this parameter as approximation: I have found different values in the literature, regarding "size of receptive field", "size of dendritic field", "center size", "visual acuity", etc, and their exact relation to each other.

To implement the simulation of the retinal representation of the image, proceed as follows:

From the selected fixation point, find the largest distance to one of the four corners of the image. Break this distance down into 10 intervals. Using those intervals, create 10 corresponding radial zones around the fixation point. For each zone, we want to find the corresponding filter: we can do so by taking the mean radius for each zone [in pixel]. From this we can find the corresponding eccentricity on the fovea [in mm], using the geometry from the figure above. This eccentricity leads to the size of the receptive field [in arcmin], which in turn can be converted into pixel, again using the geometry shown above. Selecting the next largest odd number create a symmetric filter with that side-length, and choose the filter coefficients such that they represent the corresponding DOG-filter.

Cells in V1

Activity in V1 can be simulated by Gabor filters with different orientations. For this exercise, first only use Gabor filters which respond to vertical lines.

Since I have not been able to find explicit information about any dependence of receptive field size on distance from fovea, please assume a constant receptive field size.

Input is the original image, not the input from the ganglion cells! This is due to the definition of "receptive field".

Find paramters for this vertical Gabor-filter that lead to sensible results, as assessed by visual inspection of the resulting image.

When this works, repeat this for the activity of Gabor cells with a few different orientations (0 - 30 - 60 - 90 - 120 - 150 deg), to get a "combined image" in V1.

Tips

Python

gabor_demo.py uses OpenCV to give a nice interactive example of how the output of different cells in V1 corresponds to different features of the image.

Don't forget to check out the IPYNB notebooks on image processing, which should provide a good introduction to image processing with Python.

Interesting Links

Optical illusions, the Lena Story etc.

General comments

Name the main file Ex3_Visual.py .

For submission of the exercises, please put all the required code-files that you have written, as well as the input- & data-files that are required by your program, into one archive file. ("zip", "rar", or "7z".) Only submit that one archive-file. Name the archive Ex3_[Submitters_LastNames].[zip/rar/7z].

Please write your programs in such a way that they run, without modifications, in the folder where they are extracted to from the archive. (In other words, please write them such that I don't have to modify them to make them run on my computer.) Provide the exact command that is required to run the program in the comment.

Please comment your programs properly: write a program header; use intelligible variable names; provide comments on what the program is supposed to be doing; give the date, version number, and name(s) of the programmer(s).

To submit the file, go to "Ex 3: Self-Grading".

Implementation

The whole procedure can be divided into two tasks: Task 1 and Task 2.

Read in the data

Before implementing the simulations of task 1 and task 2, we need to read in the pre-defined parameters and select the input image. The process is implemented using Python GUIs.


# Load the pre-defined parameters.
dir_crt = os.getcwd()
Params = utils.Params_visual(path_options=os.path.join(dir_crt, 'options.yaml'))
    
# GUI.
Params = utils.gui_params(Params=Params)

# Select the input image.
layout = [[PSG.Text('Select the original image (Browse or type in directly).')], 
          [PSG.InputText(), PSG.FileBrowse('Select Image')],
          [PSG.OK(), PSG.Cancel()]]
window = PSG.Window('Select the input', layout=layout, keep_on_top=True)

while True:
    event, values = window.read()
    if event in (None, 'OK'):
        # User hit the OK button.
        break
    elif event in (None, 'Cancel'):
        # User hit the cancel button.
        sys.exit()
    print(f'Event: {event}')
    print(str(values))
 
window.close()
img_input = plt.imread(fname=values['Select Image'])
if len(img_input.shape) == 3:  #  Color image
    img_input = rgb2gray(img_input)
Params.size_img_ori = np.flip(np.array(img_input.shape))
img_input = cv2.resize(img_input, dsize=tuple(Params.size_img_default), interpolation=cv2.INTER_LINEAR)

In the experiment, we used the Lena image (Figure 3) as the input.

Task 1

In task 1, we simulate the activity in the retinal ganglion cells. The whole process is :

Calculate the largest distance from the fixation point to the four corners.

Divide the distance into several intervals and create circular zones.

Determine one DoG filter for each circular zone separately.

Apply the DoG filters to the input image.

We implemented the code using a separate function. The code is:


def fun_task_1(img_input, Params):
    """Task-1 in Exercise-3.
    Task-1 is about the simulation of the activity in the retinal ganglion cells.
    The whole process is:

    Parameters
    ----------
    img_input: The input image in grayscale. Numpy array.
    Params: A class containing the pre-defined parameters.

    Returns
    -------
    response_ON: Response reaction of ON ganglion cells.
    response_OFF: Response reaction of OFF ganglion cells.
    """

    # Visualization of the input image.
    fig, ax = plt.subplots(dpi=200, figsize=(8, 6))
    plt.imshow(
        X=cv2.resize(src=img_input, dsize=tuple(Params.size_img_ori) ,interpolation=cv2.INTER_LINEAR),
        cmap='gray'
        )  #  Display in grayscale.
    plt.title(label='The original image.')
    fig.canvas.mpl_connect('button_press_event', onclick)
    plt.show()

    # Return the fixation coordinates.
    dir_crt = os.getcwd()
    coords_fixation = np.load(os.path.join(dir_crt, 'coords_tmp.npy'), allow_pickle=True)
    # Resize
    coords_fixation[0] = coords_fixation[0]*Params.size_img_default[0]/Params.size_img_ori[0]
    coords_fixation[1] = coords_fixation[1]*Params.size_img_default[1]/Params.size_img_ori[1]

    # Calculate the farthest distance to the corner and divide the distance into several intervals.
    coords_interval, zones = coord2interval(
        coords_fixation=coords_fixation, 
        img_input=img_input, 
        Params=Params
        )
    
    # Apply the appropriate DoG filter to each level.
    img_output = np.empty_like(img_input)
    for i_zone in range(1, Params.num_interval + 1):
        # Calculate the distance between the fixation point and the interval mid-point.
        dist_tmp = np.linalg.norm(coords_fixation - coords_interval[i_zone-1, :])
        # Apply the DoG filter.
        filter_DoG = get_DoG_filter(img=img_input, dist_pix=dist_tmp, Params=Params)
        img_tmp = signal.convolve2d(in1=img_input, in2=filter_DoG, mode='same')
        idx = (zones == i_zone)
        img_output[idx] = img_tmp[idx]
    
    # Simulate the ON-OFF reaction.
    # ON cells.
    response_ON = np.empty_like(img_output)
    response_ON[img_output > Params.t_ganglion_ON] = 1
    response_ON[img_output <= Params.t_ganglion_ON] = 0
    # OFF cells.
    response_OFF = np.empty_like(img_output)
    response_OFF[img_output > Params.t_ganglion_OFF] = 0
    response_OFF[img_output <= Params.t_ganglion_OFF] = 1

    # Resize to the original scale.
    response_ON = cv2.resize(src=response_ON, dsize=tuple(Params.size_img_ori), interpolation=cv2.INTER_LINEAR)
    response_OFF = cv2.resize(src=response_OFF, dsize=tuple(Params.size_img_ori), interpolation=cv2.INTER_LINEAR)

    # Visualization of the ON-OFF reaction.
    fig, ax = plt.subplots(dpi=200, figsize=(16, 6))
    plt.subplot(1, 2, 1)
    plt.imshow(X=response_ON, cmap='gray')  #  Display in grayscale.
    plt.title(label='Responses of ON ganglion cells.')
    plt.subplot(1, 2, 2)
    plt.imshow(X=response_OFF, cmap='gray')
    plt.title(label='Responses of OFF ganglion cells.')
    fig.canvas.mpl_connect('button_press_event', onclick)
    plt.show()

    return response_ON, response_OFF

Task 2

In task 2, we simulate the activities in V1. The whole process is:

Calculate the Gabor filters of different orientations.

Apply them to the input image separately.

Combine the results and take the average.

The function of task 2:


def fun_task_2(img_input, Params):
    """Task-2 in Exercise-3.
    Task-2 is about the simulation of the activities in V1.

    Parameters
    ----------
    img_input: The input image in grayscale. Numpy array.
    Params: A class containing the pre-defined parameters.

    Returns
    -------
    img_output: The output image in grayscale. Numpy array.
    """

    # First, only use Gabor filters which correspond to vertical lines.
    filter_gabor_vertical = get_Gabor_filter(theta=180, Params=Params)
    img_vertical = signal.convolve2d(in1=img_input, in2=filter_gabor_vertical, mode='same')
    img_vertical[img_vertical<0] = 0

    # Then, try different orientations for Gabor filters.
    thetas = np.arange(0, 181, 30)
    img_output = np.empty(shape=(img_input.shape[0], img_input.shape[1], len(thetas)))
    for i_theta in range(len(thetas)):
        filter_gabor_tmp = get_Gabor_filter(theta=thetas[i_theta], Params=Params)
        img_output[:, :, i_theta] = signal.convolve2d(in1=img_input, in2=filter_gabor_tmp, mode='same')
    img_output = np.mean(a=img_output, axis=2)
    img_output[img_output<0] = 0

    # Resize to the original scale.
    img_vertical = cv2.resize(src=img_vertical, dsize=tuple(Params.size_img_ori), interpolation=cv2.INTER_LINEAR)
    img_output = cv2.resize(src=img_output, dsize=tuple(Params.size_img_ori), interpolation=cv2.INTER_LINEAR)

    # Visualization of the output image.
    fig, ax = plt.subplots(dpi=200, figsize=(16, 6))
    plt.subplot(1, 2, 1)
    plt.imshow(X=img_vertical, cmap='gray')  #  Display in grayscale.
    plt.title(label='Only use the Gabor filter with vertical lines.')
    plt.subplot(1, 2, 2)
    plt.imshow(X=img_output, cmap='gray')
    plt.title(label='Use the Gabor filters in different orientations.')
    fig.canvas.mpl_connect('button_press_event', onclick)
    plt.show()

    return img_output

Example

Here we present an example of the whole process of simulations. The original input image is Figure 3.

Pre-defined parameter setting:

Select image:

The input image (grayscale):

Task 1

Circular zone division:

Responses of ganglion cells after applying DoG filters:

Task 2

Simulation results of V1 activities:

About Me

Sat, 21 Oct 2023 00:00:00 GMT

Shuo Li (李碩)

Shuo Li (李碩) is a master student in ETH Zürich, Switzerland. His major is Biomedical Engineering, Bioimaging track.

From 2011 to 2017, Shuo Li spent 6 years to finish his middle and high school career in Beijing No.12 High School. He was awarded 3 consecutive years of the Merit Student in Municipal Level. And then he was awarded the Merit Student of Beijing, which was in Provincial Level. What’s more, he won the first prize in the Beijing Mathematics Competition in 2015.

After finishing the high school career, Shuo Li attended the National College Entrance Examination of China in the summer of 2017. Then he chose SPIOEE (the School of Precision Instruments and Opto-Electronic Engineering) of Tianjin University to start his college career. During his 4 years of college career, he selected Electrical Engineering as his major and took various courses in different fields, for example, Optical Systems, Electrical Engineering, Statistics, Mechanical Engineering and Computer Science. What’s more, he also focused on the research of computer vision and signal processing. He was awarded the Merit Student of Tianjin University and the corresponding scholarship for three consecutive years. In 2018, he served as the sports minister of the students’ union in SPIOEE. By the end of his senior year of college, he was awarded the title of Outstanding Graduate of Tianjin University.

In 2021, Shuo Li started his master career in ETH Zürich. His major is Biomedical Engineering. During his master career, he chose a broad scale of courses, finished the corresponding course projects and also conducted one semester project of Neural Architecture Search (NAS) in Implicit Neural Representation (INR) models in the Computer Vision Lab (CVL). In addition, he also finished one research project of using rPPG technologies for heart rate estimation in the Biomedical and Mobile Health Technology Lab (BMHT). Currently, he will start his Master Thesis in Hong Kong. In estimation, he will finish his master degree in April, 2024.