Sunday, 30 July 2017

Tech Megatrends in 2017 - The bigger Picture

Problem: What are the major technology changes that are going to shape the enterprise over the next 5-10 years? 

Initial Hypothesis: Working within technology and making businesses more competitive I see the speed at which technology changes.  Working with multiple large customers, I know that people are generally looking at a handful of strategic technology trends.  While technology trends are actually all connected and a hierarchy.  Basically, I am classifying the significant tech trends and not drilling down to lower levels such as Mobile application patterns or particularly technologies such as O365.

While technologies are not so kind as to make themselves easy to segregate I feel the big trends are:

Analytics - Storage, mining, analyzing data and reporting.  Basically, this is an old industry with new trends such as larger data sets, new data sets e.g. social media, additional reporting formats/media, AI is really trended recognition on steroids.  The trend has been to make analytics available closer to real time and not it's moving into predictive.  Robotics is a closely related working with data. 

Security - We have so much more data and devices, the old world of protecting your own assets and monitoring outside people in a regulated fashion is no longer the main way of securing.  Key is understanding that identity is king.  Be this an individual or a computer; we need to be able to know the person or system are whom they say they are (non-repudiation).  Block-chain - While I think it's important and in some industries absolutely critical, this to me falls under security.  We need to trust and share between machines and transactions. Once again an old industry with a shift in focus in that identity and collaboration is now central to security and not using a centralized castle like security models.

Cloud Computing - Low-cost computing paid for on demand is simply a continuing trend for businesses, only now we are good at virtualization and cloud computing and the big three (AWS, Azure & Google) are getting better quickly.  Cloud computing ties to Analytics and security. Virtualization has progressed and we are getting better at providing computing safely and at a lower cost.

Sub Trends:

Robotics/Automation - Automation goes manual steps on behalf of a human and historically this has been more around manufacturing.  Going forward I see it taking over rudimentary information worker roles.  Ingesting legislative data is a great example and coupled with machine learning, we can build up models to identify what laws are applicable in different jurisdictions. Medical diagnosis are often referred to under AI is not intelligence, it's pattern recognition and analytics to determine a likely problem from the data picked up and cleaned.  The medical example will be essential to us thriving on this planet; doctors will only get complex issues, the rudimentary stuff will disappear and be dealt with better meaning fewer returning patients.

IOT - not exactly new just we have to do it better on a larger scale.  So while technology is changing, systems communication is not.  I see drones more as a subset of IOT, hardware devices using analytics.  Pretty similar to fully automated self-driving cars.

Augment Reality (AR)/Virtual Reality - will be big but like block chain will have unique application is different industries.  It relies on Analytics and security as its underpinning.

Here is some research I gleaned from 5 key firms on current technology mega-trends (this is how I see these firms views on the tech mega trends,):

PWC
===
Artifical Intelligence (AI)
Augment Reality (AR)
Blockchain
Drones (UAV)
IOT
Robotics
Virtual Reality
3D printing

EY
===
Artificial Intelligence - Cognitive learning
Robotic Automation (RPA)
Blockchain
Analytics
Internet of Things (IOT)
Cybersecurity

Deloitte
=======
Dark Analytics
Machine Intelligence
Mixed Reality
Inevitable architecture (cloud computing)
Everything as a service (cloud computing)
Blockchain

KPMG
====
Big Data & analytics/AI
Cloud Computing
Cryptocurrency/Digital Payments
IOT
Robotics
CyberSecurity
Virtual and augmented reality - stock and glasses -Google glasses

McKinsey
=======
AI
Analytics
Robotics

Possible Resolution:  Most people in technology are aware of the mega trends, how relatively important each trend is and the details of how quickly change is happening.  For me analytics, speed to market and trust need to underpin everything I deliver.

Friday, 23 June 2017

IE11 cannot save meta data when inside an iFrame

Problem:  I have two SharePoint sites, I am iFraming once site collection within a PageViewer control on the other SharePoint site.  I have disabled the iFrame security restrictions.  When I try edit the list item field columns, the data will not save in IE11.  Using IE 11 on the Order pages (without the iFraming) works.
The error manifests in that the Title is validating as if the title is blank when it's already filled in and most of the metadata is not saved/persisted on save.  MMS term data is the only data saved.

Initial Hypothesis:  As Chrome and lower level browsers work, the issue has to be with the implementation of IE.  I can see console errors in the IE developer toolbar "attachEvent" withi the SharePoint init.debug.js file is throwing the console JavaScript error. It appears that the issue only occurs in IE 11 when editing or adding list field data when the SharePoint site is within an iFrame.   So not common scenario but if you are using iFrames, it will affect a large portion of my user base.

Resolution:  Edit the master page in the Orders site collection to ensure the init.js uses the alternate path so the code shall save.


Posts on the iFraming issue Series:
IE11 not saving field data iframe Issue (This post)
Term Store/MMS musing when using iFrames (WebTaggingDialog)
Displaying SharePoint Site Pages in iFrames

Tuesday, 20 June 2017

Term store mussings with iFrames

Problem: I have a SharePoint site (ClientOrders i.e. https://orders.radimaging.com/sites/client1) being iFramed in another SharePoint site (Portal i.e. https://portal.radimaging.com/), to iframe in the ClientOrders site collection.
I want to iframe in the list within the Portal page, show the list for my customers orders and allow the user to select a different term set when editing the order metadata i.e. Language.

Initial Hypothesis:  In the site Order Site Collection, I need to allow the site to be iFramed, I do this by suppressing the iFrame HTTP header that stops sub domain iFrame using the server control: AllowFraming.  See this post for details.  Once the pages on the Order site collection can be iFramed, I use a Page Viewer Web Part in the Portal site collection on a page and now I can see my Orders inside the Portal SharePoint site collection.  All is good up to this point.

The user edits the order item metadata and when the reach the language they with to change from English to Spanish.  The user clieck on the select termset icon (two tickets) and the popup (webTaggingDialog.aspx) fails to load with the Security exception.  This aspx page uses it's own master page on the 15 hive than can be updated (watch out for CU's overriding the change).  The information below highlights the change should you be prepared to change system hive files:

When adding term set meta data to a SharePoint list item, the SharePoint use a Root hive/15 hive file, WebTaggingDialog.aspx:
/_layouts/15/WebTaggingDialog.aspx?Field=Content%20Language&IsDlg=1

Resolution: In my case, I need to allow the Order Site Collection page to be iFramed so I had to go to the WFE's and change the hive (as this is a share system page), I changed the master page to suppress the SameOrigin HTTP response (using theAllowWebFraming server control).  I added the X-Framing-Options header back with the SameOrigin and allowfrom https://order.radimaging.com using IIS.  So just plain horrible.  The WebTaggingDialog.asp does not use the site collection master page.  I amend the dialog.master file.  the page now opens but it does not load the treeview control used to display the term set data e.g. Language

Problem: Now the problem is that the WebTaggingDialog is not loading the treeview control to display the term set.

Initial Hypothesis: JS lazy loading issue or it is using the iFrame parent page (Portal) rather then (the calling page) to build up the display.

Workaround:  I don't want to mess with the root hive dialog.master or webTaggingDialog.aspx as these are system pages and I need to write custom JavaScript and inject it to make the tree control work.  I simply uses JavaScript to hide the term store popup page (I never open the webTaggingDialod.aspx).
In the end, I disabled the Term set picker control by hiding the icon (2 tickets image shown above) and type ahead works for selecting content types and I do not offer the WebTaggingDialog.aspx.

Posts on the iFraming issue Series:
IE11 not saving field data iframe Issue
Term Store/MMS musing when using iFrames (This post)
Displaying SharePoinnt Site Pages in iFrames 

Friday, 9 June 2017

SharePoint on-premise reporting options

Overview:  As always this really depends on the circumstances but my broad view on reporting for SharePoint is outlined below.

Thoughts:

  • Power BI on-prem. has not been release but will work for SSRS reports and have the Power BI functionality that shall be embedded using an iFrame into SharePoint.
  • The upgrade path from SSRS SharePoint integrated mode is going to be hard, whereas the upgrade path from SSRS native mode will be simple.
  • Power-BI on-prem is not going to support SAML.  Use WAP (Web Application Proxy)/reverse proxy to get NTLM ot Kerbros tickets for authentication.
  • SSRS SharePoint integrated mode shall be available only in SP2013 and SP2016 and not have any continued path going forward.


SharePoint 2013 Options:

SharePoint 2013 BI Options SharePoint 2013 SQL BI Options
Excel Services
PerformancePoint
SharePoint KPI/Filters
SSRS SharePoint Integrated Mode
Power Pivot for SharePoint

The table below shows what you client probably uses historically and the challenge is to map them out to a supportable solution on SharePoint going forward.

BI in SharePoint and the Microsoft stack has drastically changed over the past 2 years.  The next 12 months should keep this pace up with the release on Power BI.  As my knowledge in this area improves I'll update this post.

Other Posts that may be useful:
A good post to read as of EOY 2016 on BI  for SharePoint
SP2013 with SSRS 2012 SharePoint integrated Mode Overview
Installing SSRS on SP2013 Reporting Post

Monday, 5 June 2017

Enforcing Authentication using the Windows IdP

Overview: SharePoint farms often have multiple Identity Providers (IdP) such as Ping or SiteMinder and you always need to have Windows for the Search and other reasons.  Often you need to force the browser or CSOM code to use a specific IdP, in my case, the Windows IdP needs to be used.

In Fiddler, I have added a header as shown below to enforce the browser uses the Windows IdP:


CSOM HTTP Header added to force usage of the Windows IdP:

Simple Code for a CSOM SharePoint Test Rig



Thursday, 25 May 2017

Xamarin is fantastic-

Overview:  I am enjoying working with Xamarin, easy for any C# developer with a little experience and forethought to build decent applications.  Microsoft/Xamarin really reduces the barriers and makes building native mobile applications very achievable.  Nothing to do with SharePoint but maybe my next out of hours project will encompass O365.  I saw a demo at an Azure conference 2 months ago and thought it looks pretty straight forward.

Proposed Solution:  Scratch card game

  • Build a Web  Service for holding game and user details
  • Build UI to reveal scratch card for Android and iOS
  • Sign in using Facebook
  • Retain users winnings and games played information 

Summary: PoC is going well, deployed the Web Service and SQL backend, registered with Facebook and can login.  Secured mobile apps Droid and iOS.

Tips: PCI projects and Xamarin forms projects do not play nicely together.  Choose wisely.  Also get some help from a Xamarin developer, it is saving me so much time when i hit a problem.

Friday, 28 April 2017

Switch Master Page Minimum Permissions

Problem: Use the Client Side Object Model (CSOM C#) to add a new master pages to a site collection and switch the master page.

Initial Hypothesis: Writing to a site collection only required contribute rights or even "designer" rights at the web application permission level.

Resolution: The minimum permission set for changing master pages is "Full Permission" which a site owner and the site collection admin have.  So to switch master pages you need a high set of permissions.  UI allows master pages to be switched when the user only has "Design" permissions. This proof is flawed as the UI and CSOM permissions are different.  Can the UI have different permissions to the CSOM API???  Am I going mad.  
SPWeb object with Design user permissions cannot be updated and the API returns an "Access Denied Error" - Thanks to Sachin Khade for identifying this.

Updated 26/05/2017:  So the reply I got from the engineer who raised a Microsoft ticket is "SharePoint designer and  SharePoint GUI only need to have design permission to change the master page. This is because SharePoint designer is created as an extension of the SharePoint product. However, since CSOM calls are coded using Visual studio, the code flow involved in this is different and hence requires permissions that are higher than what SPD needs."

Summary: "Design" rights allow the user to change the master page using the UI however the same user cannot switch the master page using the CSOM C# approach.

Updated 26/05/2017: Thanks to Aswin Bhaskaran for working out a minimum permission set for using CSOM to switch the master pages on a site collection:
Note: "Design" rights can be applied at the Web Application Policy level allowing the accounts with "Design" rights the ability to add master pages.  The "Design" permission is only built into SP at the Site Collection level, I created the "Design" permission with the same permissions at the web application level to ensure my account in the Web app Design group has access to all site collections on my web app.

Note: Microsoft do not recommend customized master pages for O365 or future development.  Rather inject JavaScript to modify pages.

Thursday, 6 April 2017

SharePoint Search to extract SharePoint list Data into SQL

Problem:  I have multiple lists that I need to get data changes updated into SQL on multiple site collections.  Think 1000 site collections with 5 lists in each so 5K lists are being updated (my actual requirement is much larger).

Initial Hypothesis: In SP 2010 or SP 2013 I would use a Full Trust Event Receiver and register each list using a feature but now we are in the new world of no full trust code.  The 1st thought is RER (Remote Event Receivers), I need to register 5,000 of these and they are notoriously unreliable for delivery.  Search has to pick up all changes to list items for indexing and I can break into the crawl pipeline during Content Enrichment.

Proposed Solution:


Points to Consider:
  1. I need to create a Content Enrichment Web Service (CEWS) that I shall call from the crawl component.  I can only register a single web service on a crawl SSA so consider using the toolkit/Microsoft CEWS Framework on the Web Service endpoint this allows multiple Web Servies to be strung together.  Good idea to implement a toolkit so multiple pieces of logic can be applied.  So if BA Insights registered, then can't add another CEWS Web service. CEWS toolkit provides a pipeline of stages to add custom logic later.  So write our own stages and hook into the CEWS framework.
  2. CEWS does not process item deletes so a possibility is using the crawl log to identify deletes
  3. There is no CEWS on SharePoint Online/O365.
Note: Properties returned from CEWS has minimal data input (common custom fields such as Title) and need to augment with registered MP's to get the data in the CEWS web Service.  Do not get a dump of properties/need to register and ask for the properties.

27/04/2017 Note: An idea I had to deal with "Deletes", I could put the item into a status of "About to Delete" and the crawl would pick up the status and the CEWS could delete from he SQL database and then the Web Service could delete the list item from the SharePoint list.  It doesn't work, the CEWS pipeline does not pickup and item updated and then deleted.  So the crawl is smart but I can't enhance using this approach so more work to fix the delete ...


Wednesday, 5 April 2017

CSOM for Deployment of SharePoint sites and Assets


Problem:  Historically we built Site Templates and WSP to build up of SharePoint solutions.  With the current state of SharePoint it's not advisable to release Full Trust Code (FTC).

Initial Hypothesis: So I often tend to write CSOM for custom provisioning and asset deployment.  I utilise the Tenant Admin API for provisioning site collections and then CSOM SharePoint in C# to provision and deploy my assets.

Possible Resolution:
PnP has a great library to provision assets to base you solution on, customising and building XMLfiles allows you to quickly build repeatable SharePoint solutions that are easily configurable.
Also InstantQuick has a solution called IQApp that is worth a look at.


Saturday, 25 March 2017

Displaying SharePoint site pages in an iFrame

Posts on the iFraming issue Series:
IE11 not saving field data iframe Issue
Term Store/MMS musing when using iFrames (WebTaggingDialog)
Displaying SharEPoinnt Site Pages in iFrames (this post)

Problem:  I need to display a SharePoint 2013 document library within an iFrame of another SharePoint 2016 site.  SharePoint by default does not allow iFrame itself.

Hypothesis: IE and the other browser respect different rules for allowing iframing.  The iFraming rules are dictated by http headers output from SharePoint.  SP 2010 and MOSS did not have iFraming restrictions.  Furthermore, chrome and safari do not follow the X-Frame-Options http output but use CSP to control ancestor framing.

Resolution:
1.> Remove the SharePoint generated X-Frame-Options header generated by SharePoint - step 2 below.
2.> Add the custom X-Frame-Options http header to allow a specific domain to iframe the site (keep the attack surface as small as possible) - step 1 below.

3.> Add the Content-Secuirty-Policy frame-ancestors entry to ensure Chrome and the other browsers will allow framing as shown below.

Posts on the iFraming issue Series:
IE11 not saving field data iframe Issue
Term Store/MMS musing when using iFrames (WebTaggingDialog)
Displaying SharEPoinnt Site Pages in iFrames (this post)

Tuesday, 21 March 2017

Create a Lookup column from a list in a separate web

Problem:  I have a site collection (SPSite) with multiple sites (SPWeb) and I have a reference list in the SPSite (root web) that I want to use from multiple lists in the sites (SPWeb).

Resolution:  I recorded this short video that shows how to create a lookup site column in the site collection and then at the list within a sub web, you can associate the lookup.

video





Wednesday, 15 February 2017

MMS hybrid between SP2013 and O365 and SP2016 farms

Problem:  A lot of large enterprise customers have the Management Meta Data Service including the Content Type Hub that SharePoint farms subscribe to.  You are on-prem. with this centralised MMS and CTH.  Now you want search to work on your O365 public tenant and to use SP2016 on-prem.  It may even be more complicated with SP2016 installed on Azure and there is no direct access onto the on-prem SP2013 CTH.


Initial Hypothesis:  You want to have a central production MMS service that all SP farms subscribe to,  You can't subscribe from a SP2016 farm to the SP2013 central MMS service.  O365 can't subscribe to a different MMS, you need to use the MS MMS and sync the term store using CSOM or a tool that shall use CSOM.  Crossing domains such as in a DMZ that does not allow inbound connections look at chaining for CTHub solutions.

Restoring MMS to another farm also see moving the MMS database (think Prod for development workstations) is straight forward if you merely want another copy of the MMS, use the Export-SPMetadataWebServicePartitionData to get the MMS info and then import the MMS proxy using the PS Import-SPMetadataWebServicePartitionData.  Best post is here on exporting and importing ensuring GUIDs are maintained.  Andrew Connell has a great series on MMS and one of his post looks at the Copying the MMS instance from Prod to Development.  A normal backup and restore of the Content Hub Site Collection how to bring a copy of the CTHub back.


In Progress....

Saturday, 14 January 2017

Performance Testing SharePoint

Problem: Once again performance testing has raised caused concerns on a project.  There are various methods for calculating how many users a system can deal with.

Description:

Non-function Requirements are key to determining how "performant" the SharePoint farm needs to be to deal with peak loads.

Load testing allow us to mimic various users and see when the site/farm performance starts to degrade.  A good idea is to identify all the possible actions the users will perform  and items like Search are far more resource intensive than clicking on a link in.

Average visits per hour = (5,000 average visitors/day) / 10 hours = 500
Page Request per hour = (Ave visits/hr * 5 ave page request/hr) = 500*5  = 2,500
An example can be further broken down as follows by assuming of the users 5 request, 3 are for pages, 1 is a search and the last is viewing a document.
Recording this scenario with wait times provides a basic load test whereby the user numbers can be increased at 5 minute intervals.

WIP...