ElasticSearch + Logstash works, but does not displays any data












1















I have an Oracle DB. Logstash retrieves data from Oracle and puts it to ElasticSearch. And everything looks fine, but no changes occur on the Logstash server, as if it doesn't know what to do.



logstash.conf:



input {
jdbc {
jdbc_driver_library => "C:JBosswildfly...ojdbc7.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_connection_string => "jdbc:oracle:thin:@3d-ztemtis-ora.iba:1521/ORCL"
jdbc_user => "sample_user"
jdbc_password => "12345"
jdbc_validate_connection => true

# once a 2 minute
schedule => "2 * * * *"
statement => "SELECT * FROM table_one"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "tableone"
document_id => "%{uid}"
}
stdout{
codec => rubydebug
}
}


Logstash logs



D:Workspace3ElasticLogstashlogstash-6.5.1>binlogstash -f logstash.conf
Sending Logstash logs to D:/Workspace3/ElasticLogstash/logstash-6.5.1/logs which is now configured via log4j2.properties
[2018-11-28T00:49:30,296][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-11-28T00:49:30,308][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.1"}
[2018-11-28T00:49:33,174][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-11-28T00:49:33,455][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2018-11-28T00:49:33,471][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-11-28T00:49:33,625][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-11-28T00:49:33,674][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-11-28T00:49:33,674][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-11-28T00:49:33,699][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-11-28T00:49:33,718][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-11-28T00:49:33,745][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-11-28T00:49:33,940][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x64e24d22 run>"}
[2018-11-28T00:49:33,971][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2018-11-28T00:49:34,217][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}


ElasticSearch log



[2018-11-28T00:36:06,492][DEBUG][o.e.a.ActionModule       ] [px9stLj] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-11-28T00:36:06,683][INFO ][o.e.d.DiscoveryModule ] [px9stLj] using discovery type [zen] and host providers [settings]
[2018-11-28T00:36:07,188][INFO ][o.e.n.Node ] [px9stLj] initialized
[2018-11-28T00:36:07,188][INFO ][o.e.n.Node ] [px9stLj] starting ...
[2018-11-28T00:36:07,387][INFO ][o.e.t.TransportService ] [px9stLj] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2018-11-28T00:36:10,500][INFO ][o.e.c.s.MasterService ] [px9stLj] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2018-11-28T00:36:10,500][INFO ][o.e.c.s.ClusterApplierService] [px9stLj] new_master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-11-28T00:36:10,585][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [px9stLj] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2018-11-28T00:36:10,585][INFO ][o.e.n.Node ] [px9stLj] started
[2018-11-28T00:36:10,921][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [px9stLj] Failed to clear cache for realms []
[2018-11-28T00:36:10,962][INFO ][o.e.l.LicenseService ] [px9stLj] license [852e276a-f99f-4ce3-a5d6-86c7769ae24e] mode [basic] - valid
[2018-11-28T00:36:10,970][INFO ][o.e.g.GatewayService ] [px9stLj] recovered [3] indices into cluster_state
[2018-11-28T00:36:12,366][INFO ][o.e.c.r.a.AllocationService] [px9stLj] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[blog][0]] ...]).


As I said, the problem is - nothing is happens and no errors logged.
How can I know is this succesfully connected to Oracle?










share|improve this question























  • Run the logstash and elasticsearch in debug mode and see the logs

    – Ijaz Ahmad Khan
    Nov 27 '18 at 22:02











  • @IjazAhmadKhan Well, I'll check it out. But may be you know, why do Logstash server waits over 20 minutes before it makes any call to DB?

    – Kiryl Aleksandrovich
    Nov 27 '18 at 22:16











  • Are u sure the logstash is using ur logstash.conf or something else

    – Ijaz Ahmad Khan
    Nov 27 '18 at 22:27
















1















I have an Oracle DB. Logstash retrieves data from Oracle and puts it to ElasticSearch. And everything looks fine, but no changes occur on the Logstash server, as if it doesn't know what to do.



logstash.conf:



input {
jdbc {
jdbc_driver_library => "C:JBosswildfly...ojdbc7.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_connection_string => "jdbc:oracle:thin:@3d-ztemtis-ora.iba:1521/ORCL"
jdbc_user => "sample_user"
jdbc_password => "12345"
jdbc_validate_connection => true

# once a 2 minute
schedule => "2 * * * *"
statement => "SELECT * FROM table_one"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "tableone"
document_id => "%{uid}"
}
stdout{
codec => rubydebug
}
}


Logstash logs



D:Workspace3ElasticLogstashlogstash-6.5.1>binlogstash -f logstash.conf
Sending Logstash logs to D:/Workspace3/ElasticLogstash/logstash-6.5.1/logs which is now configured via log4j2.properties
[2018-11-28T00:49:30,296][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-11-28T00:49:30,308][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.1"}
[2018-11-28T00:49:33,174][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-11-28T00:49:33,455][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2018-11-28T00:49:33,471][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-11-28T00:49:33,625][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-11-28T00:49:33,674][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-11-28T00:49:33,674][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-11-28T00:49:33,699][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-11-28T00:49:33,718][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-11-28T00:49:33,745][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-11-28T00:49:33,940][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x64e24d22 run>"}
[2018-11-28T00:49:33,971][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2018-11-28T00:49:34,217][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}


ElasticSearch log



[2018-11-28T00:36:06,492][DEBUG][o.e.a.ActionModule       ] [px9stLj] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-11-28T00:36:06,683][INFO ][o.e.d.DiscoveryModule ] [px9stLj] using discovery type [zen] and host providers [settings]
[2018-11-28T00:36:07,188][INFO ][o.e.n.Node ] [px9stLj] initialized
[2018-11-28T00:36:07,188][INFO ][o.e.n.Node ] [px9stLj] starting ...
[2018-11-28T00:36:07,387][INFO ][o.e.t.TransportService ] [px9stLj] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2018-11-28T00:36:10,500][INFO ][o.e.c.s.MasterService ] [px9stLj] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2018-11-28T00:36:10,500][INFO ][o.e.c.s.ClusterApplierService] [px9stLj] new_master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-11-28T00:36:10,585][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [px9stLj] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2018-11-28T00:36:10,585][INFO ][o.e.n.Node ] [px9stLj] started
[2018-11-28T00:36:10,921][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [px9stLj] Failed to clear cache for realms []
[2018-11-28T00:36:10,962][INFO ][o.e.l.LicenseService ] [px9stLj] license [852e276a-f99f-4ce3-a5d6-86c7769ae24e] mode [basic] - valid
[2018-11-28T00:36:10,970][INFO ][o.e.g.GatewayService ] [px9stLj] recovered [3] indices into cluster_state
[2018-11-28T00:36:12,366][INFO ][o.e.c.r.a.AllocationService] [px9stLj] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[blog][0]] ...]).


As I said, the problem is - nothing is happens and no errors logged.
How can I know is this succesfully connected to Oracle?










share|improve this question























  • Run the logstash and elasticsearch in debug mode and see the logs

    – Ijaz Ahmad Khan
    Nov 27 '18 at 22:02











  • @IjazAhmadKhan Well, I'll check it out. But may be you know, why do Logstash server waits over 20 minutes before it makes any call to DB?

    – Kiryl Aleksandrovich
    Nov 27 '18 at 22:16











  • Are u sure the logstash is using ur logstash.conf or something else

    – Ijaz Ahmad Khan
    Nov 27 '18 at 22:27














1












1








1








I have an Oracle DB. Logstash retrieves data from Oracle and puts it to ElasticSearch. And everything looks fine, but no changes occur on the Logstash server, as if it doesn't know what to do.



logstash.conf:



input {
jdbc {
jdbc_driver_library => "C:JBosswildfly...ojdbc7.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_connection_string => "jdbc:oracle:thin:@3d-ztemtis-ora.iba:1521/ORCL"
jdbc_user => "sample_user"
jdbc_password => "12345"
jdbc_validate_connection => true

# once a 2 minute
schedule => "2 * * * *"
statement => "SELECT * FROM table_one"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "tableone"
document_id => "%{uid}"
}
stdout{
codec => rubydebug
}
}


Logstash logs



D:Workspace3ElasticLogstashlogstash-6.5.1>binlogstash -f logstash.conf
Sending Logstash logs to D:/Workspace3/ElasticLogstash/logstash-6.5.1/logs which is now configured via log4j2.properties
[2018-11-28T00:49:30,296][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-11-28T00:49:30,308][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.1"}
[2018-11-28T00:49:33,174][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-11-28T00:49:33,455][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2018-11-28T00:49:33,471][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-11-28T00:49:33,625][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-11-28T00:49:33,674][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-11-28T00:49:33,674][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-11-28T00:49:33,699][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-11-28T00:49:33,718][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-11-28T00:49:33,745][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-11-28T00:49:33,940][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x64e24d22 run>"}
[2018-11-28T00:49:33,971][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2018-11-28T00:49:34,217][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}


ElasticSearch log



[2018-11-28T00:36:06,492][DEBUG][o.e.a.ActionModule       ] [px9stLj] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-11-28T00:36:06,683][INFO ][o.e.d.DiscoveryModule ] [px9stLj] using discovery type [zen] and host providers [settings]
[2018-11-28T00:36:07,188][INFO ][o.e.n.Node ] [px9stLj] initialized
[2018-11-28T00:36:07,188][INFO ][o.e.n.Node ] [px9stLj] starting ...
[2018-11-28T00:36:07,387][INFO ][o.e.t.TransportService ] [px9stLj] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2018-11-28T00:36:10,500][INFO ][o.e.c.s.MasterService ] [px9stLj] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2018-11-28T00:36:10,500][INFO ][o.e.c.s.ClusterApplierService] [px9stLj] new_master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-11-28T00:36:10,585][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [px9stLj] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2018-11-28T00:36:10,585][INFO ][o.e.n.Node ] [px9stLj] started
[2018-11-28T00:36:10,921][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [px9stLj] Failed to clear cache for realms []
[2018-11-28T00:36:10,962][INFO ][o.e.l.LicenseService ] [px9stLj] license [852e276a-f99f-4ce3-a5d6-86c7769ae24e] mode [basic] - valid
[2018-11-28T00:36:10,970][INFO ][o.e.g.GatewayService ] [px9stLj] recovered [3] indices into cluster_state
[2018-11-28T00:36:12,366][INFO ][o.e.c.r.a.AllocationService] [px9stLj] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[blog][0]] ...]).


As I said, the problem is - nothing is happens and no errors logged.
How can I know is this succesfully connected to Oracle?










share|improve this question














I have an Oracle DB. Logstash retrieves data from Oracle and puts it to ElasticSearch. And everything looks fine, but no changes occur on the Logstash server, as if it doesn't know what to do.



logstash.conf:



input {
jdbc {
jdbc_driver_library => "C:JBosswildfly...ojdbc7.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_connection_string => "jdbc:oracle:thin:@3d-ztemtis-ora.iba:1521/ORCL"
jdbc_user => "sample_user"
jdbc_password => "12345"
jdbc_validate_connection => true

# once a 2 minute
schedule => "2 * * * *"
statement => "SELECT * FROM table_one"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "tableone"
document_id => "%{uid}"
}
stdout{
codec => rubydebug
}
}


Logstash logs



D:Workspace3ElasticLogstashlogstash-6.5.1>binlogstash -f logstash.conf
Sending Logstash logs to D:/Workspace3/ElasticLogstash/logstash-6.5.1/logs which is now configured via log4j2.properties
[2018-11-28T00:49:30,296][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-11-28T00:49:30,308][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.1"}
[2018-11-28T00:49:33,174][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-11-28T00:49:33,455][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2018-11-28T00:49:33,471][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-11-28T00:49:33,625][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2018-11-28T00:49:33,674][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-11-28T00:49:33,674][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-11-28T00:49:33,699][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-11-28T00:49:33,718][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-11-28T00:49:33,745][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-11-28T00:49:33,940][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x64e24d22 run>"}
[2018-11-28T00:49:33,971][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2018-11-28T00:49:34,217][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}


ElasticSearch log



[2018-11-28T00:36:06,492][DEBUG][o.e.a.ActionModule       ] [px9stLj] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-11-28T00:36:06,683][INFO ][o.e.d.DiscoveryModule ] [px9stLj] using discovery type [zen] and host providers [settings]
[2018-11-28T00:36:07,188][INFO ][o.e.n.Node ] [px9stLj] initialized
[2018-11-28T00:36:07,188][INFO ][o.e.n.Node ] [px9stLj] starting ...
[2018-11-28T00:36:07,387][INFO ][o.e.t.TransportService ] [px9stLj] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2018-11-28T00:36:10,500][INFO ][o.e.c.s.MasterService ] [px9stLj] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2018-11-28T00:36:10,500][INFO ][o.e.c.s.ClusterApplierService] [px9stLj] new_master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {px9stLj}{px9stLjKSkqdyzudpK1ZhA}{bkR2txqXTn-Eo1o7-2PqEA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17058418688, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-11-28T00:36:10,585][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [px9stLj] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}
[2018-11-28T00:36:10,585][INFO ][o.e.n.Node ] [px9stLj] started
[2018-11-28T00:36:10,921][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [px9stLj] Failed to clear cache for realms []
[2018-11-28T00:36:10,962][INFO ][o.e.l.LicenseService ] [px9stLj] license [852e276a-f99f-4ce3-a5d6-86c7769ae24e] mode [basic] - valid
[2018-11-28T00:36:10,970][INFO ][o.e.g.GatewayService ] [px9stLj] recovered [3] indices into cluster_state
[2018-11-28T00:36:12,366][INFO ][o.e.c.r.a.AllocationService] [px9stLj] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[blog][0]] ...]).


As I said, the problem is - nothing is happens and no errors logged.
How can I know is this succesfully connected to Oracle?







elasticsearch logstash






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 27 '18 at 21:54









Kiryl AleksandrovichKiryl Aleksandrovich

319




319













  • Run the logstash and elasticsearch in debug mode and see the logs

    – Ijaz Ahmad Khan
    Nov 27 '18 at 22:02











  • @IjazAhmadKhan Well, I'll check it out. But may be you know, why do Logstash server waits over 20 minutes before it makes any call to DB?

    – Kiryl Aleksandrovich
    Nov 27 '18 at 22:16











  • Are u sure the logstash is using ur logstash.conf or something else

    – Ijaz Ahmad Khan
    Nov 27 '18 at 22:27



















  • Run the logstash and elasticsearch in debug mode and see the logs

    – Ijaz Ahmad Khan
    Nov 27 '18 at 22:02











  • @IjazAhmadKhan Well, I'll check it out. But may be you know, why do Logstash server waits over 20 minutes before it makes any call to DB?

    – Kiryl Aleksandrovich
    Nov 27 '18 at 22:16











  • Are u sure the logstash is using ur logstash.conf or something else

    – Ijaz Ahmad Khan
    Nov 27 '18 at 22:27

















Run the logstash and elasticsearch in debug mode and see the logs

– Ijaz Ahmad Khan
Nov 27 '18 at 22:02





Run the logstash and elasticsearch in debug mode and see the logs

– Ijaz Ahmad Khan
Nov 27 '18 at 22:02













@IjazAhmadKhan Well, I'll check it out. But may be you know, why do Logstash server waits over 20 minutes before it makes any call to DB?

– Kiryl Aleksandrovich
Nov 27 '18 at 22:16





@IjazAhmadKhan Well, I'll check it out. But may be you know, why do Logstash server waits over 20 minutes before it makes any call to DB?

– Kiryl Aleksandrovich
Nov 27 '18 at 22:16













Are u sure the logstash is using ur logstash.conf or something else

– Ijaz Ahmad Khan
Nov 27 '18 at 22:27





Are u sure the logstash is using ur logstash.conf or something else

– Ijaz Ahmad Khan
Nov 27 '18 at 22:27












1 Answer
1






active

oldest

votes


















1














Please see the schedule examples here:



https://discuss.elastic.co/t/how-to-run-the-schedule-every-five-minutes-in-logstash-5-0/66222



https://www.thegeekstuff.com/2011/07/cron-every-5-minutes/



I think your schedule section should look like this:



Every 2 minutes



schedule => "*/2 * * * *"





share|improve this answer
























  • Yes, I thought "2 * * * *" is 2 minutes. Great, thank you!

    – Kiryl Aleksandrovich
    Nov 27 '18 at 22:41











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53508788%2felasticsearch-logstash-works-but-does-not-displays-any-data%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









1














Please see the schedule examples here:



https://discuss.elastic.co/t/how-to-run-the-schedule-every-five-minutes-in-logstash-5-0/66222



https://www.thegeekstuff.com/2011/07/cron-every-5-minutes/



I think your schedule section should look like this:



Every 2 minutes



schedule => "*/2 * * * *"





share|improve this answer
























  • Yes, I thought "2 * * * *" is 2 minutes. Great, thank you!

    – Kiryl Aleksandrovich
    Nov 27 '18 at 22:41
















1














Please see the schedule examples here:



https://discuss.elastic.co/t/how-to-run-the-schedule-every-five-minutes-in-logstash-5-0/66222



https://www.thegeekstuff.com/2011/07/cron-every-5-minutes/



I think your schedule section should look like this:



Every 2 minutes



schedule => "*/2 * * * *"





share|improve this answer
























  • Yes, I thought "2 * * * *" is 2 minutes. Great, thank you!

    – Kiryl Aleksandrovich
    Nov 27 '18 at 22:41














1












1








1







Please see the schedule examples here:



https://discuss.elastic.co/t/how-to-run-the-schedule-every-five-minutes-in-logstash-5-0/66222



https://www.thegeekstuff.com/2011/07/cron-every-5-minutes/



I think your schedule section should look like this:



Every 2 minutes



schedule => "*/2 * * * *"





share|improve this answer













Please see the schedule examples here:



https://discuss.elastic.co/t/how-to-run-the-schedule-every-five-minutes-in-logstash-5-0/66222



https://www.thegeekstuff.com/2011/07/cron-every-5-minutes/



I think your schedule section should look like this:



Every 2 minutes



schedule => "*/2 * * * *"






share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 27 '18 at 22:30









Ijaz Ahmad KhanIjaz Ahmad Khan

3,39611426




3,39611426













  • Yes, I thought "2 * * * *" is 2 minutes. Great, thank you!

    – Kiryl Aleksandrovich
    Nov 27 '18 at 22:41



















  • Yes, I thought "2 * * * *" is 2 minutes. Great, thank you!

    – Kiryl Aleksandrovich
    Nov 27 '18 at 22:41

















Yes, I thought "2 * * * *" is 2 minutes. Great, thank you!

– Kiryl Aleksandrovich
Nov 27 '18 at 22:41





Yes, I thought "2 * * * *" is 2 minutes. Great, thank you!

– Kiryl Aleksandrovich
Nov 27 '18 at 22:41




















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53508788%2felasticsearch-logstash-works-but-does-not-displays-any-data%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

A CLEAN and SIMPLE way to add appendices to Table of Contents and bookmarks

Calculate evaluation metrics using cross_val_predict sklearn

Insert data from modal to MySQL (multiple modal on website)