Advanced Tutorial
===================

Create HCX for CX2 network
----------------------------

Hierarchical Network Schema (HCX_) can be obtained by adding specific annotation to a network in `CX2 format`_.
HCX format allows to link the hierarchical network to its interactome.
Here we describe how to annotate CX2 network to use the Multi-file model for HCX.

1. Add following attributes to **networkAttributes**

* ``ndexSchema``
    with value ``hierarchy_v0.1``
* ``HCX::modelFileCount``
    with value of the actual count of all the files that this hierarchy references, including the hierarchy itself.

    If you link only full interactome (parent network) the value will be 2. If each node in the hierarchy points to
    its own interaction network, it will be the number of all the interactomes plus 1 for the hierarchy itself.
* one of the following attributes:
    (optional in case each node points to its own interactome, in this case a node attribute will be set, **see step 2**)

    - ``HCX::interactionNetworkName`` - filename of file containing parent network located in the same directory of the hierarchy
    - ``HCX::interactionNetworkUUID`` - uuid of parent network in NDEx
    - ``HCX::interactionNetworkURL`` - url of parent network in NDEx

2. Add following attributes to **nodes**

* ``HCX::isRoot``
    true for root node and false for the rest

* ``HCX::members``
    list of IDs of nodes in the interaction network.

* ``HCX::interactionNetworkUUID``, ``HCX::interactionNetworkName`` or ``HCX::interactionNetworkURL``
    that allow to specify individual interaction networks for each node, it overwrites the value declared in the networkAttributes aspect.
    This attributes are optional if the general interaction network was specified in networkAttributes.

    When specifying individual interaction networks for each node, ``HCX::memberNames`` attribute can be used instead of ``HCX::members``.
    ``HCX::memberNames`` should have a list of node (gene) names, as in CD_memberList attribute in the hierarchy network generated by the CDAPS app in Cytoscape.

Other node attributes are optional, more information available in HCX_ specification.

Example:

.. code-block:: python

    import ndex2
    from ndex2.cx2 import RawCX2NetworkFactory

    # Create NDEx2 python client
    client = client = ndex2.client.Ndex2(username='<USERNAME>', password='<PASSWORD>')

    # Create CX2Network factory
    factory = RawCX2NetworkFactory()

    parent_net = factory.get_cx2network('path_to_your_interactome_in_cx2')
    hier_net = factory.get_cx2network('path_to_your_hierarchy_in_cx2')

    # Save interactome (parent network) to NDEx to get the url with uuid
    parent_url = client.save_new_cx2_network(parent_net.to_cx2(), visibility='PUBLIC')

    # Add HCX related attributes to networkAttributes
    hier_net.add_network_attribute('ndexSchema', 'hierarchy_v0.1', datatype='string')
    hier_net.add_network_attribute('HCX::modelFileCount', '2', datatype='integer')
    hier_net.add_network_attribute('HCX::interactionNetworkUUID', parent_url.split('/')[-1], datatype='string')

    # Look for roots and add HCX::isRoot attribute to nodes
    all_nodes = set(hier_net.get_nodes().keys())
    targets = set()
    for edge_id, edge_obj in hier_net.get_edges().items():
        targets.add(edge_obj['t'])
    # Source node is not a target of any edge
    root_nodes = all_nodes.difference(targets)

    attr_name = 'HCX::isRoot'
    for node_id in hier_net.get_nodes().keys():
        hier_net.add_node_attribute(node_id, attr_name, str(node_id in root_nodes).lower(), datatype='boolean')

    # Add HCX::members attribute to nodes
    for node_id, node_obj in hier_net.get_nodes().items():
        memberlist = hier_net.get_node(node_id).get('v', {}).get('CD_MemberList', '').split(' ')
        membersids = []
        for member in memberlist:
            membersids.append(parent_net.lookup_node_id_by_name(member))
        hier_net.add_node_attribute(node_id, 'HCX::members', membersids, datatype='list_of_integer')

    print(hier_net.to_cx2())

    # Save hierarchy to NDEx
    client.save_new_cx2_network(hier_net.to_cx2(), visibility='PUBLIC')

.. _HCX: https://cytoscape.org/cx/cx2/hcx-specification/
.. _`CX2 format`: https://cytoscape.org/cx/cx2/specification/cytoscape-exchange-format-specification-(version-2)

Modify network attributes
--------------------------

Network attributes in NDEx can be programmatically modified to update or correct information. This involves downloading
the network from NDEx, processing the attributes, and applying transformations or replacements as needed.

Using regular expressions, specific patterns within attribute values can be identified and replaced to maintain data
integrity or reflect updated resources. After modifications, the network can either be saved as a new instance or
updated in place on the NDEx server.

Here’s the process:

- Connect to NDEx: Authenticate using a username and password.
- Download the Network: Retrieve the desired network in CX2 format from NDEx using its UUID.
- Create a Network Object: Use the ``RawCX2NetworkFactory`` to convert the downloaded network into a ``CX2Network`` object.
- Modify Attributes: Iterate through network attributes and make updates.
- Save or Update the Network: Save the modified network as a new CX2 network (recommended for safety) or overwrite the existing network.

Similar process can be applied to modify node or edge attributes.

Example: Replace broken image URLs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: python

    import re
    import json
    import ndex2
    import io
    from ndex2.cx2 import RawCX2NetworkFactory
    from ndex2.client import DecimalEncoder

    # Create NDEx2 python client
    client = ndex2.client.Ndex2(username='<USER>', password='<PASSWORD>')

    # Create CX2Network factory
    factory = RawCX2NetworkFactory()

    # Download network which network attributes you want to modify
    client_resp = client.get_network_as_cx2_stream('17a6b7ce-b342-11ef-99aa-005056ae3c32')

    # Convert downloaded network to CX2Network object
    net = factory.get_cx2network(json.loads(client_resp.content))

    # Define the pattern to be replaced and the replacement (here to replace broken URLs with new ones, converting NCBI URLs to EuropePMC)
    pattern = r'https://www.ncbi.nlm.nih.gov/pmc/articles/([A-Za-z0-9]+)/bin/([^"]+)'
    replacement = r'https://europepmc.org/articles/\1/bin/\2'

    # Iterate through attributes and replace defined pattern in each
    for attribute_key, attribute_val in net.get_network_attributes().items():
        # Apply a regex pattern to replace broken URLs with new ones (in this case, converting NCBI URLs to EuropePMC)
        modified_attribute_val = re.sub(pattern, replacement, attribute_val)
        # Set modified attribute value for given attribute key
        net.add_network_attribute(attribute_key, modified_attribute_val)

    # Save network to NDEx, value returned is link to raw CX data on server.
    res = client.save_new_cx2_network(net.to_cx2(), visibility='PRIVATE')

    # Update network in place (WARNING: it will override original network - use with caution)
    # # Create bytes stream
    # stream = io.BytesIO(json.dumps(net.to_cx2(), cls=DecimalEncoder).encode('utf-8'))
    # # Update network in NDEx by completely replacing the network with
    # # one set in cx_stream
    # client.update_cx2_network(stream, 'aad56606-b342-11ef-99aa-005056ae3c32')