As a user, I can see my keyword raw result

[rosle]

What happened

✅ Setup sass
✅ Run the background job to get the result from google search - experiment with GenServer
✅ Update the keyword status and result

Insight

For this one, I am trying to setup a background job to search. A GenServer + Task Supervisor is setup to handle that.

There are still some known problems that I need to investigate and fix later. 😓

1. With lots of keyword (tried with around 100+ keys), some is not processed.
2. Not sure how the supervisor behaves if we run lots of keywords at the same time.
3. On the test part is still weird, I put some sleep to let the task finish before checking the result.

Proof Of Work

• After upload the keyword, it shows the status on the list

  

• The status is updated after the keyword search is done

  

• User can click on the view_result to view the search result

  

[olivierobert]

How/when does handle_info gets called? 🤔 start_task is started without any links so how?

[rosle]

For this one,

it happens when we start the task with Task.Supervisor.async_nolink below. it will send the message with task reference and the result back here.

But if the task is failed, it will trigger the handle_info({:DOWN, ref, :process, _pid, _reason}, state) below.

More info:
https://hexdocs.pm/elixir/Task.Supervisor.html#async_nolink/3

[olivierobert]

I understood more by seeing the actual code example.

At first, I was confused as I thought that the supervisor would be notified, not the Genserver. For the supervisor to be notified, then it requires to use start_child.

[olivierobert]

But if this process dies then all state would be gone, right?

When restarting should it then reads from the database to pick up all keyword remaining to be scraped? 🤔

[rosle]

Yep true. I haven't think about this case 😂

[olivierobert]

I noticed that you used the same technic to inject the right module based on the environment (as per this blog post I shared). I even feel that this is something we could bring to Ruby projects

[rosle]

Another interesting ideas -> https://medium.com/flatiron-labs/rolling-your-own-mock-server-for-testing-in-elixir-2cdb5ccdd1a0 🤔 This one is to running a server in the test env.

[olivierobert]

How about adding the sleep in the Google API Client Mock to simulate delays?

[rosle]

I think adding the delay to the client, we will need to put here also because we need to wait for the task to complete. (api is called in the task)

I found there is another way that we can monitor if the task is actually finish -> https://stackoverflow.com/questions/28745193/testing-asynchronous-code-in-elixir/28746505#28746505

Which I can do something like

task = SearchKeywordWorker.search(keyword.id)

ref = Process.monitor(task.pid)
assert_receive {:DOWN, ^ref, :process, _, :normal}, 500

But because even if the task is completed, I do some db update in the genserver handle_info and I encounters the error

I'm not completely sure what it is but maybe they're trying to update(in the genserver) and get the record(in the assertion). So when I use that, I added that I had to put antoher sleep before the assertion to make it work

ref = Process.monitor(task.pid)
assert_receive {:DOWN, ^ref, :process, _, :normal}, 500
:timer.sleep(100) 

🤔 💭

[olivierobert]

It's actually a good idea to use a behaviour so make sure that both the actual and mock modules have the same implementation.