APS11: Analyzing Real-time Internal User Searches

Nick DeNardis: So, everyone coming to your site, your institution site, any site in general is not just a user. They’re coming there for purpose and that’s why you need to think of them as an individual and not as just another number.
 
So, a little bit about myself. I’m the Associate Director of Web Communications at Wayne State University in Detroit. I host a video blog called EDU Checkup and I write for a little blog called .eduGuru. 

What I am going to talk to you today about is Driving Decisions with Data. So my background is on Computer Science and so I love data. I love analyzing data, I love everything about it because seeing things come in and analyzing things that are actually real definitely has more value than just kind of making decisions on a whim. And so that’s what we try to do as much as possible at Wayne State. We try to stir not only like individual decisions about certain design elements, information elements or how we program something. Not just based on opinion but also based on real data.

Often, it’s the only thing that can cut through a committee or cut through a big echo on campus, is showing data of real users and real environments to really just bring into the picture that although you may think one thing, there are others who are definitely think differently.

So why does an institution have a website? They really, you know, this has been talked about a lot at the conference. Why is the website even needed? It’s definitely not for the designs. So students aren’t coming to your institution to look at the design of your site. It’s definitely not for the photos. You’re going to have tons of photos on your site and they’re not coming there to just browse photos. They’re coming to your site to look for information. They’re coming there for on a specific purpose and they know that purpose whether they know it consciously or not. They’re going there to find out some piece of information. And so we know that. As Web publishers, we know that we’re publishing information for our visitors to consume and for them to have a successful visit at our site.  But how come this happens over and over and over again?

So if you’re not familiar with this xkcd cartoon, over on the left hand side, it’s "Things On The Front Page of A University Website". Now, the right hand side is "Things People Go To The Site To Look For". The only two in common is the full name of the university.

So administrators and the people, I want to say the people in charge because it sounds a little weird because as Web professionals, technically, we’re in charge of what gets published on the site. The motivation of the people behind those decisions, unfortunately aren't always the motivation of the visitor. And so, although we think that 'Campus Photo Slideshow' and 'Letter From The President' are important, students are really coming to find 'Parking Information'. They're going to find the 'Usable Map' and things like that.

So, it’s like a little bit about our website. So this is our site. It’s nothing special at all. It was designed a few years ago. It was designed by a committee. It does its job and we work with it. That’s just where we’re at right now. And so what happens—let’s go and do some numbers here. So for the past… This is a one year segment of our numbers. So, from September 10th, 2009 till 2010. So in total, in that timeframe, we had 8.7 million visits, an those are individual visits, not necessarily visitors and most, the majority of them, are internal visitors. But we started looking at some other elements within the site. So the actual search is things that people search for on our homepage. We found that the majority of them almost 700,000 are coming from those external group. So in total, set 11% use of all the visitors used the search on our site but 25% are from external visitors. So, we started thinking. Wow, this is a lot of information that people are searching for on a regular basis. What are they searching for?

So, we used the standard tools. Google Analytics, Google Webmaster tools, CrazyEgg, kind of see where people are clicking on our site; and then we started playing. We actually got in a beta of Woopra. I don’t know if you had experience with Woopra before but it’s a real time analytics for your site. So you can see who’s on there, where they came from, where they went, how many times they had been there before and how long they stayed in every single page. It really gives an insight into a visitor that personalizes it beyond just some number or aggregate of how many visits this page got at the end of the day or end of the week. But even after using all these tools we weren’t happy. We still felt like there was something going on on our Search that we didn’t know about. So what we ended up doing is, well, we decided, we wanted to figure out what every visitor story is and the only way to do that is to analyze every visitor to figure out what they were actually doing and figure out what we weren’t providing that they had a search for almost a million times in a year

So, this going to go OK. So every visit starts with the first impression. Whether external or internal visitor, you have different motives, you still go through the same 10-second task. So you figure out, is the site going to have what I’m looking for, am I confident in its ability to give me the information and in a timely fashion. So, is the capability of this page whether be the homepage or any child page, any page beyond the homepage going to have the information that I’m looking for?

So, external visitors are pretty easy. They notice the visuals. They orient themselves with the navigation. They scan and if they’re disappointed, they use the Search. They’re used to it. Google has turned everybody into a Google culture. Why bother searching through pages and navigation items and things like that when you can just search for it and within that first page, you’re going to get something pretty useful. We didn’t have that ability before Google was around. Well, we kind of did but it wasn’t inherent in to us.

Then, the internal visitor. So, it’s another day, it’s another dollar. I need to go find XYZ. I’ve never seen it in the main navigation. I don’t have time to explore. I just need to get this information so they just search for it and they know that they’re pretty confident something will come up.

So, this is kind of what we found that we came up with as far as some searches, that would come up kind of not over and over but really pick our attention. So, what is Wayne State's zip code? Well, it's on the footer of every page and someone is searching for it. Why isn't it apparent to this person? 

Print all my application I already filled out. So, us, as Web workers, we know once you start filling out an application, you get to a status page and that’s where you can find out your status. You can print your application. That’s not always apparent to our visitors.

Let’s see, like 2,000 on cash and tuition expenses. These are important things that people are searching for that would never come up in a Google Analytics report just because they maybe just one search. They maybe worded 500 different ways, 500 different times and it would still come up as one single record within Google and we couldn’t make that correlation.

What happens if I don’t apply for graduation? Well, you won’t graduate. How do I prepare for a career fair? There’s probably good information out there about the stuff. We just don’t think about it when we’re producing content.

So, this is really annoying, it's really slow. So, this is kind of what we see from our Top 10 search terms in Google—pipeline, just student portal housing job, campus mappings, like that. These come up often and we struggle with this because, yes, these are pretty prominent in our site. People still search for them. They just kind of do it inherently. They don’t want to look at navigation items.

And so what we did is, internally, we kind of added a script to our Search box on our top level homepage, plus kind of the 50 to 70 kind of child pages within that. We didn’t edit every single page on every single site because that would be way too much information. Just to kind of see where, what people are searching for and what page they’re on that they search for. And so it would intercept every average single search that came in, dump it into the database and we would start. We created a tool that just showed to us in real time. So as people were searching for things, they would start showing up on our screen. What we did is we just added it to our secondary monitor. I had everybody in my department do that, just passively look at it all day long.

And so we started developing, noticing some patterns. And so people search for existing sites a lot. A lot of times, they search for housing, they search for financing aid, they search for registrar’s office, they search for the academic calendar.  This happens on a day to day basis and we know those items are really easy to find but people are still searching for them and what goes in to that search is an actual search, submit, go to a secondary page with the results and they find the result and then they go off to their actual page that they were really looking for.

Same thing with people. So people are looking for other people whether they be faculty or staff and an expected pattern is we have a lot of people searching for address on our 'Contact Us' page and that really kind of stump us. Because the address, our location is right in the middle of the page, and people were still searching for address. And we still have not been able to figure that out why people are doing this and it comes up every so often. It’s not like everyday but there are simple things like that that we see over and over. 

So we started categorizing every single result. So, we wanted to kind of get an idea of what these searches were about. So we went through – well, it was mainly me – almost a million results and I tried to categorize almost all of them. So 20% came out to users completely rough numbers because I didn’t get through every search term. Twenty-eight percent departmental or office sites; 21% program requirements, programs or requirements; 15% of faculty or staff members; 12% of being location stuff, so either an individual building or where to send like transcript or something like that; and then 24% other. Just random stuff that could have been, I don’t even know, like of the hours of the residence hall being open or the dining hall being open.

So what we decided was we need to get a quick fix out there. We weren’t happy with someone searching for departmental site, jumping to a search results page and then getting them to their site that they’re looking for. So we decided, we took two or three hours and we implemented auto suggest on our search.

So we have a site index. We started modifying our site index to include every site that we knew, our main departmental sites, and then we had an official title with them and then a whole list of ‘also known as’ titles. So that we know the Recreation and Fitness Center as Recreation and Fitness Center but students may think of it as just the gym.

And so we started categorizing all of our official site index thing so that when someone starts typing in like 'Calendar' on our search, becomes both to drop down of academic calendar, admissions calendar. And as someone they can use their mouse or they can use the keyboard, go down, click on one of those items and it goes directly to those site. No need for search results page.

So, after implementing this, we found that 35% of users started using auto. So, Auto Select.

Audience: Thirty-five percent of what users?

Nick DeNardis: Thirty-five percent of users that searched.

Audience: OK.

Nick DeNardis: So, of all of the searches that were happening, 35% of them we funneled directly to the departmental site or the school and college site or like academic calendar. So we have some special pages out there that we knew people were looking for. So, this directly skip to that results page. So, it took one last step for users to find information they were looking for.

So, second example: first week of classes, we started seeing spikes of these terms – class schedule, campus map, academic calendar, bookstore, and driving directions. These things we found out through the search that were really important. We kind of knew that students needed these information anyway but we didn’t really know until we started seeing how many people were searching for these items like it’s a big Web traffic data first week of classes. And so these searches just kind of flew in. We had at least four, five a second that we’re coming in here. So our institution is 33,000 students about and most of them – we're urban university – so most of them commute. We only have maybe 10% of students that live on campus.

So we did is… This does not work here. All right, so I don’t think I have another picture of this. What we did is, this is… What was that?

Audience: Can you turn the light off up there?

Nick DeNardis: I don’t know what I can do. What I’ll do is… Let me skip back to our homepage real quick. So what we did is in this space here, we could have just taken over the homepage with these few items because we knew that this wasn’t a Recruitment Day. It wasn’t a day that students were coming to look for Open House information. We could have taken most of the space away from our traditional homepage and put this really important resources out there, but we couldn’t really do that. And so what we did is we took this section here, we moved it down and we put a big bar down here for 'First Week Resources'. And we put these items right down on the page so that someone coming to the site could just quickly click on one and not have to use the search. We thought this was a pretty good idea. 

So, this is actually a hit map of the site with those elements on it. You could see a couple of them were starting to get quite a few clicks and actually what… It was kind of my fault. I sent this up in CrazyEgg for 10,000 visits and we ended up using them all in three hours, in almost four hours.

[Laughter]

Nick DeNardis: And so it was a pretty good representation of how many people click on it but within those three house, we ended up with over, a little, almost 11,000 clicks. It sounds weird but because we also track. We have movable elements on the page so clicks can… Don’t necessarily mean that someone's going to jump to another page and we also track non-active clicks. So, if you click on the background element or something that looks like a link but really isn’t a link, those also get tracked.

So anyway, in the week, we got quite a few clicks to these elements. Actually, we had a lot of good feedback from students that said that they were surprised that we were kind of taking the initiative to give them things that we necessarily, that we're kind of forward thinking in what they were needing. So to us, that was really important and without seeing this kind of real-time search results, we would not have been able to do this. We would look at our Google Analytics report the next day or, I think it’s updated per hour or something like that, and we would have kind of gasp, "Oh look! A lot of people are searching for a driving directions, " which doesn’t usually come up on a day to day basis because once you know how to get to campus, it’s pretty much the same way everyday. And so we would have been a day late or a few days late. To us, this was really important to show that we are really reacting to students' needs. Unfortunately though, we got a lot of clicks on our homepage. This is only 0.13% of all the clicks on the homepage but it showed us that this information in a timely manner was definitely important and we only had this up for a week – September 1st through 7th.

So for us, the typical search experience is kind of cumbersome as it currently sets. What we’ve been learning through looking at each of these search terms, through kind of mining through all these data, really, what it comes down to is that we need to make changes but we can’t make drastic changes. So we’re trying to move. We’re using this data to move our site in a direction that is more useful one thing at a time. In that way we can test each thing individually and then we can see how it overall impacts the search results, what people are searching for and how they searching for it and at the same time kind of keep movement on the site.

So we weren't planning on doing something like running a six-month study, doing a final 20-page report and what we learned and then having some action items, putting in together a committee. We weren’t concern with any of that. These things were what were important to us – moving the site in a incremental direction as we went on and learning from those different changes.

So this is kind of how our 'Search' looks right now. Someone does a search for 'Computer Science', it goes to just kind of iFrame with Google Custom Search inside of it. It looks really bad. It’s really kind of clunky. The search results don't start to like halfway down the site. We don’t really have a lot of control over what is in there because we don’t have a Google Search Appliance so we work with what we have. What we do have is other databases of important information that these students are looking for. Some of that information is Events, some of it is Classes, some of it is Building Information, some of it is Faculty Staff or other Student information. So what we decided to do is change the Search page. We’re actually using Google custom Search API, I believe, to pull in Web results.

So these are the same Web results in a little bit skinnier column. But then on the side here, we’re using that the space to put in. So if you search for Computer Science, the same search has events that are going on that are either Computer Science related, in the title, have the tags on it. Computer Science classes, that come up; the buildings which Computer Science classes are being held in. And then below here, which actually I didn’t add, is Faculty and anybody who is teaching in Computer Science.  

And so what we’re doing is taking a page which right now is really clunky. Because we learned that these are the most popular categories of things that people search for regardless of how they search, because we can’t control that, we started to add this kind of really, I guess, compartmentalized and easy to digest areas where they can get information as quickly as possible. Because for us as Web publishers, that is what they’re looking for and that is how… No matter how they get there, we need to get them there as quickly as possible because we can’t dictate how they use our site. They’re used to using Google. They’re used to using tools that find them information or they're used  to finding information to the non-official channels, I guess you would say. 

All right. So, as we go through this, I just want to bring up, this is our blog that we write about all of these changes on. So, this presentation actually be up on a blog after this talk and you’ll be able to see actual…

So now it’s your turn. Something that I unfortunately did not get a 100% done before this presentation but I’m going to have it done pretty soon, is the ability for you guys to be able to do this on your site. So it’s something that I have kind of started right now. The site is queryspy.com. What is there right now is just kind of a sign-up page for me to get people who are interested in looking at search results on your site in real-time.

So I’m not exactly sure how this is going to develop but I know that it was really important for our department to see this information and I know if you watch EDU Checkup, if you read .eduGuru or anything that I’ve been a part of, I’m a huge fan of sharing – sharing data, sharing information – because we can all learn from each other. We’re all in similar situations and this type of information is so crucial to those one-person departments that have zero budget and they run a ton of stuff. Finding this information out is definitely in their best interest and it can make their lives better. I have had at least a dozen people tell me that after their review, like doing the review of their site on EDU Checkup, that push their administration to give them more resources. So it’s really kind of a misunderstanding but it’s more of getting the understanding what is required of a Web department to fully function to give a good experience to your users. It takes resources and this data is definitely going to help that

So, data driving decision is what it is all about. We collect as much data as we can as a department. Although we don’t look at every single piece of it, we know it’s there. It’s at our fingertips and if we need to analyze something, we analyze it and start watching. If you can start watching that data come in, the better and more prepared you’ll be to either defend or make changes and then the better prepared you’re going to be when you’re actually implementing any Web solutions because you have an idea of the context in what your producing. 

So I just want to get two shoutouts to this Higher Ed Analytics that Karine's doing which is really cool. It’s a bunch of higher ed institutions who are sharing their base analytics on a monthly basis, and so basically crowd sourcing, average clicks through rates to applications, average homepage bounce rate, things like that. That although you see it for your own institutions, seeing it at a broader sense it gives a better understanding of the picture. Then, I always follow Trending Upward which is a really good blog about sharing analytics, well, higher ed analytics. 

All right, and that’s my talk.  If any of you guys have questions, feel free to ask. And slides and everything will be up on our Web Communications blog after this. 

Yes? 

Audience: What were you using to get to your own results side by side with Google? Is that some Javascripts up there? 

Nick DeNardis: Yeah. So this page is…

Audience:  Those look like search terms, somehow.

Nick DeNardis: Yes, So this page actually is… So Google Custom Search allows you to have… So there’s a couple of ways that you can display the results and one of them is actually just do an AJAX call and so we’re taking this search. This is actually put in here with Google’s API. So it’s just like one line of Javascript and it like puts that Search box on your site and then you just give it, I think, an ID of an area to dump all the results into; and so we’re just dumping it into this area here. 

This page, unfortunately, it doesn’t look as pretty without Javascript on, and so that’s why this hasn’t been launched yet. Also, there’s some style issues that we’re just kind of not working on yet because the functionality is not there 100% but we do have a few users using this that we trust to basically analyze how useful this is. And so yeah, Google is actually kind of pulling that just with an AJAX request in shoving it unto a div. 

Audience: In Google, when you’re on a stop, did you just pull search terms from the URL?

Nick DeNardis: Yes. It’s using one Javascript request and actually each of this elements load in separately. So although we have all this information like in kind of shadow databases that are really close to this page so that it’s not hitting like super live data. It’s actually pulling it live. 

Yes? 

Audience: I looked at the Google Search’s terms of use with the years of… It precluded displaying other search results alongside Google results.    

Nick DeNardis: It’s a good question. I don’t know.

[Laughter]

Nick DeNardis: So that... 

Audience: Oh, I’m not taking it… I  don’t know if that’s… I know what you’re saying but I think they would allow this.

Nick DeNardis: Yeah.

[Cross-talk]

Well, if you don’t tell them, I won’t. And So...

[Laughter]  

Audience: There you go. That’s pretty common functionality in the Google Search Appliance and...

[Cross-talk]

Nick DeNardis: OK. That’s something that I need to look into then and so I would probably be writing about that on the blog. 

Audience: We do it as well

Nick DeNardis: Oh you do it also? Alright.

Audience: We did it for a couple of years.  

Nick DeNardis: If no one said anything then, I think we’re safe. Unless you’re getting like a 100,000 search terms every minute, you’ll be fine. Yeah, they have a lot more to worry about. 

Yes?

Audience: How does the Auto-Select work? Does that when...? 

Nick DeNardis: So yeah, the Auto-Select is actually… We did a jQuery plug-in called Auto-suggest. And so what it does is you just give it like an idea of your actual info box and it will send it to a page as an AJAX request and then you do whatever processing you want on your end and send back JSON to the actual script. And it will format it in like what title and URLs and stuff like that. So you can actually even manipulate the URLs in there that have like Google campaign code or something. You know, like have those types of tracking inside of those clickers.  

Yes? 

Audience: What kind of strategies are you looking at... major change in URL?

Nick DeNardis: Yeah, that’s a tough one. So that’s why we have like a student minion for. So every week, one of his recurring tasks is to go to the site Index page and use like the link checker Firefox plug-in to make sure all of those links work. There’s, I believe, I want to say like 3,500 links on there. So that’s one of his task, test every week. 

Audience: Thirty five hundred links?

Nick DeNardis: No. There’s a Firefox plug-in called like link checker or something. And it just like highlights any other ones that aren’t working in red. 

Audience: Is that a pinger? 

Nick DeNardis: Pinger? I don’t know. There’s probably a lot. 

Audience: ...in red.

Nick DeNardis: OK. So pinger is the plug-in. No, he doesn’t physically click them. That would be horrible. 
Yes? 

Audience: When you use those specialized links during registration, like the driving direction or things like that? 

Nick DeNardis: Yeah. 

Audience: Do you get a lot of complains? Can you think of some out of hand?

Nick DeNardis: We haven’t heard of any complains.

Audience: Really?

Nick DeNardis: So, if people are complaining they’re not telling us. So we monitor Twitter quite a bit and Facebook and everything like that. So people aren’t complaining when they’re gone and I think because we labeled them 'First Week Resources'. I think that kind of yielded that they’re not going to be here forever. So I think we used that wording on purpose so that people didn’t think that it was a permanent feature of the homepage. 

Yes? 

Audience: I heard you mentioned you have a student portal? 

Nick DeNardis: We do have a student portal. I don’t know if I mentioned it. But ...

Audience: There's something about Pipeline or so…

Nick DeNardis: Yes, Pipeline is our student portal. 

Audience: Right. Under the circumstances.

Nick DeNardis: Yes. 

Audience: My question is...

Nick DeNardis: Yes.

Audience: Are you carrying these across to the  portal?

Nick DeNardis: No. So…

Audience: Are you reusing any of this functionality?

Nick DeNardis: No. We are not mainly because the portal is not in our like our domain of control. So, we just have been trying to impact what we control and we’ll as we probably rule this out a little bit more to kind of the schooling college level websites because those also get a large amount of traffic. We will probably look at the portal and other pieces of software that students use on a regular basis. But they do use… That was like their number one task, when they come to our homepage, is to log in to either Pipeline, the webmail or Black Bird. So that’s probably why they’re searching for it often. So having this quick link and the auto suggestion here was a huge benefit for those people because they didn’t have to hit that intermediary page.

Any other questions? Yeah?

Audience: Are there any other times like for future editing like 'Graduation Page or anything like that?

Nick DeNardis: Yeah. So during graduation week, during registration, so like when there is… When registration starts, so like the, I think, the Winter semestral classes just went up and then maybe in a week or so, the registration will actually begin. What we’ll do is we're going to take over this whole middle section. We would just maybe… We kind use this for like recruitment and we might kind of move that down a little bit and have like a special box inside of there. Or we might… I kind of been experimenting with this but I haven’t seen, I don’t know, I’m not happy with it. If someone actually does select that Search box even without clicking, without typing anything, having a few like quick things like that that are time sensitive up in that box automatically before they actually even start typing. But when we kind of tested it internally, it just kind of felt really clunky and so we didn’t go and put that on there.

Anybody else? Kind of went faster when I thought . What I can do is actually -- what time does this go to?

Audience: Three-thirty.

Nick DeNardis:  Three-thirty? OK, we got a couple of minutes. What else we going to do is bring up the actual. So this is kind of see… You guys are actually seeing real-time, kind of what we see, and this doesn’t have all the features. So like all the categorization and stuff like that is going in a different tool than this. This will actually show…Hopefully, this is actually still working. I didn’t see anything. So like we see like that when just came in payroll. It was from an internal computer and they were searching from the actual homepage. Thanks.

[Laughter]

So, like this Wayne financial webpage. From an external person, searching for financial aid. These are the types of things that like kind of confuse us and we don’t know why people are searching for these types of things but – I think this is being weird in Chrome because it’s like showing double results – but those are types of things that we see every once in a while. So what we’ll do is like on that financial web page, we’ll like change the page and we’ll mark what we changed and on what day and then we’ll try to see if those drop off, those anomalies and searches  drop off. But we wouldn’t know about those without being able to see these.

Yeah, the next biggest thing that we have found is that like on our 'Programs' page which lists all of our degree programs, people are using the Search on the top of that page to find the program. So we haven’t been able to find a good… Well, I have been heavy with some of the designs that the designers have come up with to be able to have two search boxes on a page and have the user kind of only know that the one within the page content searches within the same programs.  So that’s our next biggest hurdle.

All right, enough of this. So yeah, all right. If there’s no other questions, I’m going to be around and if anyone has any implementation questions, I’m happy to answer them.

Yes?

Audience: I was going to say,  I think the reason people use Search is this is the only feature that’s consistent from website to website.

Nick DeNardis:  Yeah.

Audience: I mean if we had some other guys, in every single website had, people probably will use it. Everybody uses search, everybody knows how to use it.

Nick DeNardis: Exactly.

Audience: So that’s… Even if you're on the Contact, you're going to search for Events which is… It just works.

Nick DeNardis: Yeah, the thing we didn’t want to do… I don’t want this to be OK. One thing that we didn’t want to happen was somebody beyond the Contact Page looking for an address, getting to a search results page, and then going back to the Contact Us page. So those are types of things that we really found were the most detrimental to the user experience because that only frustrates them that, "I was just on this page," and they’re going to be in this never-ending loop. Yeah, that’s one thing that we definitely do not want to happen. But without seeing this data we would have no idea.

Cool. Thank you guys. 

[Applause]