Friday, January 13, 2017

Neo4j 3.1 Embedded mode GraphDatabase browser available by wrapping with NeoServer

Issue: A Neo4j 3.1 embedded graph database does not expose the jetty based http browser (7474) by default

Solution: Wrap the embedded graph database with a NeoServer
It turns out with a bit of help from the following post we can still bring up a wrapping NeoServer around our embedded mode Neo4j 3.1.0 server
http://stackoverflow.com/questions/30074232/replacement-for-deprecated-wrappingneoserverbootstrapper

We override the following function

    @Override
     protected GraphDatabaseBuilder.DatabaseCreator createDatabaseCreator(
                final File storeDir, final GraphDatabaseFactoryState state)  {
            return new GraphDatabaseBuilder.DatabaseCreator() {
                @Override
                public GraphDatabaseService newDatabase( final Map<String, String> config ) {
                    EnterpriseBootstrapper neoServer = new EnterpriseBootstrapper();
                    GraphDatabaseService graph = null;
                    // convert all config (spring, conf, code) to vararg Pairs
                List<Pair<String, String>> pairs = new ArrayList<>();
                for(Entry<String, String> entry : config.entrySet()) {
                pairs.add(Pair.of(entry.getKey(), entry.getValue()));
                } 
                    Pair<String, String> pairArray[] = new Pair[pairs.size()];
                    
                    // will resolve to /dir/data/databases/graph.db 
                    int state = neoServer.start(storeDir, Optional.empty(), pairs.toArray(pairArray)); 
                    // state is 0 for success, 1 will mean a null server
                    if(state > 0) {
                    log.error("return state of NeoServer.start(); is 1 - no GraphDatabase available - check config settings");
                    }
                    // will be null if state == 1
                    NeoServer server = neoServer.getServer();
                    if(null != server) {
                    Database database = server.getDatabase();
                    if(null != database) {
                    graph = database.getGraph(); 
                    // set the paxos HA listener only when dbms.mode=HA
                    // Note: initial TO_MASTER callback during server.start above is missed
                    if(graph instanceof HighlyAvailableGraphDatabase) {
                    HighlyAvailableGraphDatabase haGraph = (HighlyAvailableGraphDatabase) graph;
                    haMonitor.setDb(haGraph);
                    HighAvailabilityMemberStateMachine memberStateMachine = 
                    (haGraph).getDependencyResolver()
                    .resolveDependency(HighAvailabilityMemberStateMachine.class);
                    if ( memberStateMachine != null ) {
                    memberStateMachine.addHighAvailabilityMemberListener(haMonitor);
                    System.out.println("register: " +  haMonitor);
                    // rethrow isMaster callback from start
                    //haMonitor.getMasterListenerManager().masterChanged(haGraph.isMaster());
                    System.out.println("isMaster: " + haGraph.isMaster());
                    }
                    }
                    } else {
                    log.error("database null : check your http configuration settings");
                    }
                    } else {
                    log.error("server null : check your http configuration settings");
                    }
                    return graph;
                } };

    }

Using the following spring configuration

   
    <context:annotation-config />
    <context:spring-configured />
    <!-- rest annotations -->
    <mvc:annotation-driven />
    
    
    <!--  Rest controllers -->
    <context:component-scan base-package="org.obrienlabs.nbi.graph.service" />
    <!-- in cases where the DAO's are in a separate jar - list them -->
    <bean id="daoFacade" class="org.obrienlabs.nbi.graph.service.ApplicationService"/>
    <bean id="IGraphDatabaseService" class="org.obrienlabs.nbi.graph.service.GraphDatabaseServiceImpl"/>
    <util:map id="config">
        <entry key="ha.allow_init_cluster" value="true"/>
        <entry key="unsupported.cluster_name" value="NSP"/>
        <entry key="ha.host.coordination" value="127.0.0.1:5001"/>
        <entry key="ha.pull_interval" value="10s"/>
        <entry key="ha.host.data" value=":6016"/>
        <entry key="ha.slave_only" value="false"/>
        <entry key="ha.tx_push_factor" value="1"/>
        <entry key="ha.tx_push_strategy" value="fixed_ascending"/>
        <entry key="ha.server_id" value="1"/>
        <entry key="ha.initial_hosts" value="127.0.0.1:5001"/>
        <entry key="dbms.mode" value="HA"/><!-- required to get the http browser -->
        <entry key="browser.allow_outgoing_connection" value="true" />
        <entry key="unsupported.dbms.ephemeral" value="false" />   
        <entry key="dbms.connector.bolt.enabled" value="true" />
        <entry key="dbms.connector.bolt.address" value="0.0.0.0:7688" />
        <entry key="dbms.connector.bolt.type" value="BOLT" />
        <entry key="dbms.connector.bolt.tls_level" value="DISABLED" />
        <entry key="dbms.connector.http.type" value="HTTP" />
        <!-- enabled=false is not supported in this configuration -->
        <entry key="dbms.connector.http.enabled" value="true" />
        <entry key="dbms.connector.http.address" value="0.0.0.0:7575" />
        <entry key="dbms.connector.http.encryption" value="NONE" />
        <entry key="dbms.security.auth_enabled" value="false"/>
        <entry key="dbms.logs.debug.level" value="DEBUG"/>
        <entry key="dbms.logs.http.enabled" value="true" />
        <entry key="dbms.logs.query.enabled" value="true"/> <!--  causes netty exceptions when ssl enabled -->
        <entry key="dbms.shell.enabled" value="true"/>          
    </util:map>
    
    <bean id="haMonitor" class="org.obrienlabs.nbi.graph.service.HaMonitor"/>  
    <bean id="graphDbFactory"  class="org.obrienlabs.nbi.graph.service.ExtendedHighlyAvailableGraphDatabaseFactory">
        <constructor-arg ref="haMonitor" />
    </bean>
    <bean id="graphDbBuilder" 
        factory-bean="graphDbFactory" 
        factory-method="newEmbeddedDatabaseBuilder">
        <constructor-arg value="/ec2-user"/>
    </bean>
    <bean id="graphDbBuilderFinal" 
        factory-bean="graphDbBuilder" 
        factory-method="setConfig">
        <constructor-arg ref="config"/>
    </bean>
    <!-- HighlyAvailableGraphDatabase wrapped by an EnterpriseBootstrapper NeoServer created in this constructor -->
    <bean id="graphDatabaseService" 
        factory-bean="graphDbBuilderFinal" 
        factory-method="newGraphDatabase" 
        destroy-method="shutdown" />



Details:

Run the following code in
https://github.com/obrienlabs/nbi-neo4j-embedded-aws-war



Neo4j 3.1.0 HA (paxos) Embedded mode HighlyAvailableGraphDatabase wrapped with an EnterpriseBootstrapper NeoServer

In this code we get everything (except causal cluster using the raft protocol).

With a few modifications I was able to wrap a HighlyAvailableGraphDatabase with a EnterpriseBootstrapper.

There are some non-fatal exceptions around JMX reporting in debug.log likely related to my 8.0.28 tomcat version but the graph db running embedded in Tomcat is OK.

        2016-12-21 16:20:00.574+0000 INFO  Bolt enabled on 0.0.0.0:7688.
        2016-12-21 16:20:09.554+0000 INFO  Attempting to join cluster of [127.0.0.1:5001]
        2016-12-21 16:20:11.566+0000 INFO  Creating new cluster with name [neo4j.ha]...
        2016-12-21 16:20:11.607+0000 INFO  Instance 1 (this server)  entered the cluster
        2016-12-21 16:20:12.164+0000 INFO  I am 1, moving to master
        2016-12-21 16:20:12.293+0000 INFO  Instance 1 (this server)  was elected as coordinator
        2016-12-21 16:20:12.462+0000 INFO  I am 1, successfully moved to master
        2016-12-21 16:20:12.513+0000 INFO  Instance 1 (this server)  is available as master at ha://127.0.0.1:6001?serverId=1 with StoreId{creationTime=1482199697648, randomId=7800059877674392627, storeVersion=15531981201765894, upgradeTime=1482199697648, upgradeId=1}
        2016-12-21 16:20:14.495+0000 INFO  Database available for write transactions
        2016-12-21 16:20:31.917+0000 INFO  Mounted REST API at: /db/manage
        2016-12-21 16:20:53.264+0000 INFO  Remote interface available at http://localhost:7575/
        register: org.obrienlabs.nbi.graph.service.HaMonitor@1c0f80c9


Source:
https://github.com/obrienlabs/nbi-neo4j-embedded-aws-war

Diagnostics:
7575 http and 7688 bolt ports
```
obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 7575
tcp46      0      0  *.7575                 *.*                    LISTEN      131072 131072  49013      0
tcp4       0      0  127.0.0.1.7575         127.0.0.1.60685        TIME_WAIT   407296 146988  49013      0
tcp6       0      0  ::1.7575               ::1.60699              TIME_WAIT   407284 146808  49013      0
tcp6       0      0  ::1.7575               ::1.60700              TIME_WAIT   407284 146808  49013      0
obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 7688
tcp6       0      0  ::1.7688               ::1.60704              ESTABLISHED 406582 146808  49013      0
tcp6       0      0  ::1.60704              ::1.7688               ESTABLISHED 398196 146808  48165      0
tcp6       0      0  ::1.7688               ::1.60702              ESTABLISHED 406570 146808  49013      0
tcp6       0      0  ::1.60702              ::1.7688               ESTABLISHED 398185 146808  48165      0
tcp6       0      0  ::1.7688               ::1.60701              ESTABLISHED 407255 146808  49013      0
tcp6       0      0  ::1.60701              ::1.7688               ESTABLISHED 407628 146808  48165      0
tcp46      0      0  *.7688                 *.*                    LISTEN      131072 131072  49013      0
obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 8080
tcp4       0      0  127.0.0.1.8080         127.0.0.1.60584        FIN_WAIT_2  408104 146988  49013      0
tcp4     994      0  127.0.0.1.60584        127.0.0.1.8080         CLOSE_WAIT  408128 146988  42992      0
tcp46      0      0  *.8080                 *.*                    LISTEN      131072 131072  49013      0
Links:
Deployed demo is on our Canadian data center at
(bolt enabled) http browser - user:neo4j pass:password

Add a node to the embedded graph db

Neo4j 3.1 Technology

https://neo4j.com/blog/neo4j-3-1-beta-release/?ref=home
- causal clustering (with read replicas and synced read/write clusters)
- bolt can drive load balancing (with read/write splitting)
- 4 security roles with list and terminate control
- node and id reallocation from deleted records (uuid reuse?)
- max query duration

A server-mode Neo4j 3.1.1 instance is running for development purposes on 
http://ec2-52-20-143-25.compute-1.amazonaws.com:7474/browser/

Troubleshooting

./neo4j start does not work
Fix: make sure your java 8 home is set like

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home

Links




No comments:

Post a Comment