SDAP-70 Link events with dataset collections#33
SDAP-70 Link events with dataset collections#33lewismc wants to merge 3 commits intoapache:masterfrom
Conversation
|
Can one of the admins verify this patch? |
|
There is still some work to be done on the WebUI. However the data modeling should be good. I would like to demo this shortly so will keep this PR open such that I can receive and accommodate feedback. |
|
|
||
| public void startFullIngest() { | ||
| public void startFullIngest(MudrodEngine me) { | ||
| DiscoveryEngineAbstract wd = new WeblogDiscoveryEngine(props, es, spark); |
There was a problem hiding this comment.
EventIngester has a variable of ESDriver type since EventIngester extends MudrodAbstract class. I don't think you have to change function from startFullIngest() to startFullIngest(MudrodEngine me).
quintinali
left a comment
quintinali
left a comment
There was a problem hiding this comment.
@lewismc I tested your code on my local computer and it works. The event module can download EONET events and save them into Elasticsearch.
| recom.process(); | ||
|
|
||
| EventIngester eonet = new EventIngester(props, es, spark); | ||
| eonet.ingestAllEonetEvents(me); |
There was a problem hiding this comment.
I suggest removing input parameters "me" since EventIngester already has a variable of ESDriver.
| private void startFullEonetIngest(MudrodEngine me) { | ||
| LOG.info("Starting full EONET ingest."); | ||
| EventIngester eonet = new EventIngester(props, es, spark); | ||
| eonet.ingestAllEonetEvents(me); |
There was a problem hiding this comment.
I suggest removing input parameters "me" since EventIngester already has a variable of MudrodEngine.
| @SuppressWarnings("unused") | ||
| GetResult result = null; | ||
| String index = mEngine.getConfig().getProperty(MudrodConstants.ES_INDEX_NAME); | ||
| String eventType = "eonet_event"; |
There was a problem hiding this comment.
Did you add eventType to the configuration file?
| } | ||
| } | ||
| esDriver.destroyBulkProcessor(); | ||
| return ""; |
There was a problem hiding this comment.
The function will return empty string whether the index operations success or fail and made it hard to write a unit test case for function ingestAllEonetEvents.
3 participants
This issue is linked to https://issues.apache.org/jira/browse/SDAP-70