Reading a SharePoint list with Java: tutorial

Using external business data with SharePoint part 2: SharePoint Web Services and Java

In this tutorial I will show how to consume and use a SharePoint Lists Web Service from java with Netbeans. The objective will be to read the items of a SharePoint list from a java application. The functions coded in this tutorial will be needed later on if you’re trying to use external business data with SharePoint (my next post will describe how to the create/update/delete items in a list). Sounds pretty basic? Well, I haven’t found any other tutorials of this kind for Java, so enjoy!

Tutorial: reading a list

This tutorial has been tested Netbeans 6.8 and SharePoint Online, however any other Java IDE should be suitable. It should work with MOSS 2007 too (please let me know if it doesn’t). First of all, I’d recommend reading my previous post about the challenges to expect while coding such an application in Java. Before you begin, make sure you have a Sharepoint list that’s populated with a few items.

1. Get the WSDL. Before launching Netbeans, you’ll need to download the Lists web service WSDL file with your browser to a local directory. The URL to your WSDL should look like: sharepointsite.com/_vti_bin/Lists.asmx?WSDL. The reason why you’ll download the WSDL instead of parsing it remotely from your Netbeans project is because you’ll want to avoid errors at build or run time. I tried with Netbeans 6.5, 6.7, 6.8, and encountered a variety of problems when I tried to access and parse the WSDL remotely (NTLM authentication errors, jaxb parsing errors, WSDL not found errors, etc)… Parsing it locally seems to be the simplest workaround to those Netbeans bugs complexities.
2. Parse the WSDL. In order to use Sharepoints Web Services with Java, you will need to generate its stub classes with its WSDLs.
• From Netbeans (easy): Open Netbeans and create a new Java Application. Right-click on your project and create a new Web Service Client, and choose your Lists WSDL file. Choose JAX-WS as Client Style. Click finish and that’s it. To make sure everything’s fine, clean and build your project.
• From the console : If you’re not using Netbeans, you can generate the stub classes manually from the console, thanks to the JDK wsimport tool. Open the console (cmd.exe) and open the directory where your Lists.wsdl file is located. Run the following command to generate the classes in the directory (change the paths if necessary) “C:\Program Files\Java\jdk1.6.0_12\bin\wsimport.exe” -p com.microsoft.schemas.sharepoint.soap -keep -extension Lists.wsdl. Import those classes in your IDE.


3. Code the SOAP authentication function. The first coding we’ll do is to create a Manager class that will take care of the Lists Web Service authentication and manipulation. If successful, it will return an open lists port (ListsSoap object) that will offer all the functions we need to use the Lists Web Service. Here’s the authentication function:

    /**
     * Creates a port connected to the SharePoint Web Service given.
     * Authentication is done here. It also prints the authentication details
     * in the console.
     * @param userName SharePoint username
     * @param password SharePoint password
     * @return port ListsSoap port, connected with SharePoint
     * @throws Exception in case of invalid parameters or connection error.
     */
    public static ListsSoap sharePointListsAuth(String userName, String password) throws Exception {
        ListsSoap port = null;
        if (userName != null && password != null) {
            try {
                Lists service = new Lists();
                port = service.getListsSoap();
                System.out.println("Web Service Auth Username: " + userName);
                ((BindingProvider) port).getRequestContext().put(BindingProvider.USERNAME_PROPERTY, userName);
                ((BindingProvider) port).getRequestContext().put(BindingProvider.PASSWORD_PROPERTY, password);
            } catch (Exception e) {
                throw new Exception("Error: " + e.toString());
            }
        } else {
            throw new Exception("Couldn't authenticate: Invalid connection details given.");
        }
        return port;
    }

Now that the auth code is made, you may call it from your main class to make sure the connection went fine. This code will display some information in the console in case of error.

4. Code a function that’ll display XML docs in your console. Most of the time, SharePoint Web Services will return you a huge chunk of information in Microsoft’s own XML / CAML format. This is precisely what we’ll be interested in, as we’ll parse these Web Service responses later to display the lists items for instance. It’s very helpful to have such a function for debugging purposes. We’ll use the old school XML parsing tools embedded in the JDK for that job (javax.xml.transform).

    /**
     * Creates a string from an XML file with start and end indicators
     * @param docToString document to convert
     * @return string of the xml document
     */
    public static String xmlToString(Document docToString) {
        String returnString = "\n-------------- XML START --------------\n";
        try {
            //create string from xml tree
            //Output the XML
            //set up a transformer
            TransformerFactory transfac = TransformerFactory.newInstance();
            Transformer trans;
            trans = transfac.newTransformer();
            trans.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
            trans.setOutputProperty(OutputKeys.INDENT, "yes");
            StringWriter sw = new StringWriter();
            StreamResult streamResult = new StreamResult(sw);
            DOMSource source = new DOMSource(docToString);
            trans.transform(source, streamResult);
            String xmlString = sw.toString();
            //print the XML
            returnString = returnString + xmlString;
        } catch (TransformerException ex) {
            Logger.getLogger(Manager.class.getName()).log(Level.SEVERE, null, ex);
        }
        returnString = returnString + "-------------- XML END --------------";
        return returnString;
    }

5. Code a function that’ll read your lists items. This function will send a getListItems request to the lists web service using the open ListsSoap port we created before. That’s the easy part! The challenging part is interpreting the dense Soap response, which has that infamous CAML structure. Basically, you have to cast the Web Service response into a classic XML document. That way, you can parse it with your favorite XML reader once you know what you’re looking for.

    /**
     * Connects to a SharePoint Lists Web Service through the given open port,
     * and reads all the elements of the given list. Only the ID and the given
     * attributes (column names) are displayed, as well as a dump of the SOAP
     * response from the Web Service (for debugging purposes).
     * @param port an already authentificated SharePoint Online SOAP port
     * @param listName original name of the Sharepoint list that is going to be read
     * @param listColumnNames arraylist containing the various names of the Columns
     * of the SharePoint list that are going to be read. If the column name isn't
     * found, then an exception will be thrown
     * @param rowLimit limits the number of rows (list items) that are going to
     * be returned
     * @throws Exception
     */
    public static void displaySharePointList(ListsSoap port, String listName, ArrayList<String> listColumnNames, String rowLimit) throws Exception {
        if (port != null && listName != null && listColumnNames != null && rowLimit != null) {
            try {

                //Here are additional parameters that may be set
                String viewName = "";
                GetListItems.ViewFields viewFields = null;
                GetListItems.Query query = null;
                GetListItems.QueryOptions queryOptions = null;
                String webID = "";

                //Calling the List Web Service
                GetListItemsResponse.GetListItemsResult result = port.getListItems(listName, viewName, query, viewFields, rowLimit, queryOptions, webID);
                Object listResult = result.getContent().get(0);
                if ((listResult != null) && (listResult instanceof ElementNSImpl)) {
                    ElementNSImpl node = (ElementNSImpl) listResult;

                    //Dumps the retrieved info in the console
                    Document document = node.getOwnerDocument();
                    System.out.println("SharePoint Online Lists Web Service Response:" + Manager.xmlToString(document));

                    //selects a list of nodes which have z:row elements
                    NodeList list = node.getElementsByTagName("z:row");
                    System.out.println("=> " + list.getLength() + " results from SharePoint Online");

                    //Displaying every result received from SharePoint, with its ID
                    for (int i = 0; i < list.getLength(); i++) {

                        //Gets the attributes of the current row/element
                        NamedNodeMap attributes = list.item(i).getAttributes();
                        System.out.println("******** Item ID: " + attributes.getNamedItem("ows_ID").getNodeValue()+" ********");

                        //Displays all the attributes of the list item that correspond to the column names given
                        for (String columnName : listColumnNames) {
                            String internalColumnName = "ows_" + columnName;
                            if (attributes.getNamedItem(internalColumnName) != null) {
                                System.out.println(columnName + ": " + attributes.getNamedItem(internalColumnName).getNodeValue());
                            } else {
                                throw new Exception("Couldn't find the '" + columnName + "' column in the '" + listName + "' list in SharePoint.\n");
                            }
                        }
                    }
                } else {
                    throw new Exception(listName + " list response from SharePoint is either null or corrupt\n");
                }
            } catch (Exception ex) {
                throw new Exception("Exception. See stacktrace." + ex.toString() + "\n");
            }
        }
    }

6. Test the whole thing!

Everything should be ready. Let’s test it from main (of course replace the necessary values with your own).

    public static void main(String[] args) {
        try {

            //Authentication parameters
            String userName = "yourUsername";
            String password = "yourPassword";

            //Opening the SOAP port of the Lists Web Service
            ListsSoap port = Manager.sharePointListsAuth(userName, password);

            /*
             * Lists Web service parameters
             * The list names below must be the *original* names of the list.
             * if a list or column was renamed from SharePoint afterwards,
             * these parameters don't change.
             */
            String listName = "Client";
            String rowLimit = "150";
            ArrayList<String> listColumnNames = new ArrayList<String>();
            listColumnNames.add("Title");
            listColumnNames.add("otherColumnName");

            //Displays the lists items in the console
            Manager.displaySharePointList(port, listName, listColumnNames, rowLimit);
        } catch (Exception ex) {
            System.err.println(ex);
        }
    }

This is it!I hope you found this post useful. Any thoughts? I’d be happy to hear your feedback. You can download the full source code covered in this tutorial here.

Update 26 February 2010: Removed unnecessary web service URL in the authentication code (thanks Claus Fassing!)